Method, device and system for training a result prediction model, and result prediction method and device

Abstract

Disclosed are a training method of a result prediction model, a result prediction method and device. The training method comprises: obtaining a sentence containing comparative semantics as implicit evidence E ent for training; obtaining comparative content E ent and a comparative result r from the implicit evidence E dis ; and training the result prediction model according to the comparative content E dis and the comparative result r. The present application can train a result prediction model by collecting massive comparative information in natural text, and can predict the result of a behavior to be performed by widely using the effect of an existing behavior, thereby reducing the trial and error cost.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a result prediction model training method, a result prediction method and device. BACKGROUND

[0002] In the era of evidence-based medicine, various treatment methods need to be supported by corresponding evidence. The evidence often comes from high-quality clinical trials. Implementing clinical trials is time-consuming and labor-intensive, and requires a lot of resources. Clinical trials with design flaws or difficulties in success also occupy valuable patient resources, and may force the clinical trials that are urgently needed to be terminated due to the failure to recruit enough patients. The Remdesivir clinical experiment conducted during the COVID-19 epidemic is an example: due to the occupation of a lot of patient resources by other clinical trials with design flaws or difficulties in success, the experiment did not recruit enough patient resources, and did not get statistically significant results. The new clinical trial needs to be supported by the past clinical evidence. However, the past clinical evidence is in the millions, and the ability of individuals or teams to integrate the past clinical evidence is limited.

[0003] Not only clinical trials, but also other behavior choices such as fitness, scientific diet, etc. can learn from the previous similar behaviors. But through the conventional way of searching on the Internet or seeking expert advice, it is often difficult to make a good prediction of the behavior to be performed.

[0004] Therefore, there is a need for a solution that can predict the behavior result in the design stage and prioritize more effective behaviors. SUMMARY

[0005] One of the technical problems to be solved by the present disclosure is to provide a result prediction model training method and a corresponding prediction method using the model. The method trains the result prediction model by aggregating a large amount of comparative information in natural text, and can widely use existing achievements to predict the behavior result in the design stage, thereby reducing the behavior trial and error cost.

[0006] According to a first aspect of the present disclosure, a result prediction model training method is provided, comprising: obtaining a sentence containing comparative semantics as implicit evidence E ent for training; obtaining comparative content E ent and a comparison result r from the implicit evidence E dis ; and training the result prediction model according to the comparative content E dis and the comparison result r.

[0007] Optionally, the sentence containing comparative semantics is collected from natural text contained in a specific field database as implicit clinical evidence E ent for training the specific field result prediction model.

[0008] Optionally, the sentence containing the predetermined comparison template in the natural text is searched as the implicit clinical evidence E ent .

[0009] Optionally, the method further comprises reversing the comparison content E dis and the comparison result r simultaneously, thereby obtaining reverse order comparison content E rev and the reversed comparison result Rev(r), and training the result prediction model according to the reverse order comparison content E rev and the reversed comparison result Rev(r). Optionally, training the result prediction model comprises pre-training a sequence modeling model for result prediction. The pre-trained sequence modeling model for result prediction comprises a Transformer encoder whose pre-trained model parameters are initialized by BioBERT. And the pre-training process optimizes the cross-entropy between the predicted result and the actual result.

[0010] According to a second aspect of the present disclosure, a result prediction method is provided, comprising: inputting behavior data to be performed into the result prediction model obtained according to the first aspect; and obtaining the behavior result predicted by the result prediction model.

[0011] According to a third aspect of the present disclosure, a result prediction method is provided, comprising: inputting data of a test to be performed into the clinical trial result prediction model obtained according to the first aspect; and obtaining the clinical trial result predicted by the clinical trial result prediction model.

[0012] According to a fourth aspect of the present disclosure, a training device of a result prediction model is provided, comprising: a collection unit configured to obtain a sentence containing comparison semantics as implicit evidence E ent for training; an extraction unit configured to obtain comparison content E ent and a comparison result r from the implicit evidence E dis ; and a training unit configured to train a result prediction model according to the comparison content E dis and the comparison result r.

[0013] According to a fifth aspect of the present disclosure, a result prediction device is provided, comprising: a data input unit configured to input behavior data to be performed into the result prediction model obtained according to the first aspect; and a result obtaining unit configured to obtain the behavior result predicted by the result prediction model.

[0014] According to a sixth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method according to the first and second aspects described above.

[0015] According to a seventh aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method according to the first and second aspects described above.

[0016] The present solution predicts the outcome of a behavior by a large-scale implicit evidence pre-training method. In the present solution, the outcome prediction model can accept any natural language input, and thus can model a wider range of behaviors. The present solution collects large-scale unstructured implicit evidence in a heuristic manner for learning, avoiding the cost of structured explicit evidence labeling. Further, the present solution pre-trains with a comparative language model, which can better learn the comparison between different behavior choices, rather than the simple co-occurrence relationship in the language model. The present solution can achieve significant performance improvement on standard datasets. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, and in which:

[0018] Figure 1 A schematic flow chart of a method for training an outcome prediction model according to an embodiment of the present application is shown.

[0019] Figure 2 A schematic flow chart of a method for training an outcome prediction model according to an embodiment of the present application is shown.

[0020] Figure 3 A schematic diagram of a large-scale implicit evidence pre-training method is shown.

[0021] Figure 4 A schematic diagram of an outcome prediction task is shown.

[0022] Figure 5 A schematic diagram of a training device for an outcome prediction model according to an embodiment of the present application is shown.

[0023] Figure 6 A schematic diagram of a training device for an outcome prediction model according to an embodiment of the present application is shown.

[0024] Figure 7 A schematic diagram of a computing device according to an embodiment of the present application is shown, which can be used to implement the model training and outcome prediction described above. DETAILED DESCRIPTION

[0025] The preferred embodiments of the present disclosure will be described in greater detail below, with reference to the accompanying drawings. While the preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0026] In the era of evidence-based medicine, various treatment methods need to be supported by corresponding evidence. The evidence often comes from high-quality clinical trials. The implementation of clinical trials is time-consuming and labor-intensive, and requires a large amount of resources. Clinical trials with design defects or difficulties in success will also occupy valuable patient resources, and may force the clinical trials that are urgently implemented to be terminated due to the failure to recruit enough patients. The clinical experiment of Remdesivir in the COVID-19 epidemic is an example: due to the occupation of a lot of patient resources by other clinical trials with design defects or difficulties in success, the experiment did not recruit enough patient resources, and did not get statistically significant results.

[0027] The design of new clinical trials needs the support of past clinical evidence. For example, the WHO recommends that four drugs, chloroquine / hydroxychloroquine, remdesivir, interferon, and lopinavir / ritonavir, be given priority in clinical trials for COVID-19. The evidence for this recommendation is that these drugs are effective against related coronaviruses in past laboratory or human trials. However, there are millions of past clinical evidences, and the ability of individuals or teams to synthesize past clinical evidence is limited. A study found that about 86.2% of clinical trials ultimately fail. Even some COVID-19 treatments recommended by the WHO, an authoritative organization, did not have good clinical results.

[0028] Therefore, there is a need for a solution that can predict the results of clinical experiments at the design stage and prioritize clinical experiments with a higher probability of success.

[0029] To this end, the present application trains a result prediction model by aggregating a large amount of comparative information in natural medical texts, widely utilizes existing research results to predict the results of clinical experiments at the design stage, and thereby eliminates clinical experiments with poor expected results at the design stage, thereby improving the effectiveness of real clinical experiments.

[0030] To assist the design of clinical trials, the present application predicts the results of clinical trials by a large-scale implicit evidence pre-training method. In the training scheme of the present application, the result prediction model can accept any natural language input, so it can model a wider range of interventions. This scheme collects large-scale unstructured implicit evidence in a heuristic way for learning, avoiding the cost of structured explicit evidence annotation. Further, this scheme uses comparative language model pre-training, which can better learn the comparison between the treatment group and the control group, rather than the simple co-occurrence relationship in the language model. This scheme can significantly improve the results on standard datasets.

[0031] Not only clinical trials, but also other behavior choices such as fitness, scientific diet, etc. can also draw on various similar behaviors of predecessors. However, through the conventional way of network search or seeking expert advice, etc., it is often difficult to make a good prediction of the behavior to be performed. Therefore, the present application can also train a result prediction model for each specific field by obtaining corpus of each field, so as to give an expectation for the behavior of people and reduce the cost of trial and error.

[0032] Figure 1 A schematic flowchart of a result prediction model training method according to an embodiment of the present application is shown. This method can especially use existing natural language medical literature to predict the results of clinical trials, thereby guiding the design of clinical trials and avoiding unnecessary clinical trials. It should be understood that this method can also be used for training of result prediction models of other behaviors other than clinical trials. The present application does not limit this.

[0033] In step S110, a sentence containing comparative semantics is obtained as implicit evidence E ent Here, the implicit evidence can be especially implicit clinical evidence, for example, when predicting the results of clinical trials, the content obtained from the medical literature library. When predicting behaviors such as fitness and diet, the content can be obtained from professional websites or fitness platforms.

[0034] The sentence containing comparative semantics refers to a sentence in which the semantics involves comparing one object with other objects. For example, in the sentence "artemether is more effective than artemisinin in inhibiting the proliferation of MCF-7 tumor cells", there is a comparison between one object "artemether" and another object "artemisinin". Therefore, this sentence can be regarded as a sentence containing comparative semantics, and as implicit clinical evidence E ent .

[0035] In step S120, the comparison content E ent is obtained from the implicit evidence E disand comparison result r. For example, the comparison result can be extracted from the implicit clinical evidence, thus obtaining comparison content E dis and comparison result r. Here, the comparison result can be the result of comparing a certain object with other objects in a sentence containing comparison semantics, i.e., the word indicating the result semantics in the implicit evidence. For example, in the sentence “artemether is stronger than artemisinin in inhibiting the proliferation of MCF-7 tumor cells”, the comparison result can be “stronger than …” or, after determining the comparison meaning, only “strong” is taken as the comparison result r. Comparison content E dis can be taken as implicit evidence E ent the part left after removing the comparison result. For example, “artemether is stronger than artemisinin in inhibiting the proliferation of MCF-7 tumor cells []”.

[0036] In step S130, the result prediction model is trained according to comparison content E dis and comparison result r. For example, comparison content E dis and comparison result r are used to train the clinical trial result prediction model. Specifically, comparison content E dis is taken as training data, and the result prediction model can predict the correct comparison result r. For example, in the field of clinical trials, “artemether is stronger than artemisinin in inhibiting the proliferation of MCF-7 tumor cells []” is input, and the clinical trial result prediction model is trained to input the correct comparison result r, i.e., to fill in “strong” in [].

[0037] Thus, the present application trains the result prediction model by summarizing the comparison information in existing natural text, and can use the existing behavior effect to predict the behavior result in the design stage, thereby reducing the behavior trial and error cost. In the field of clinical medicine, the results of clinical trials in the design stage can be predicted by using existing clinical research results, thereby improving the trial effect of real clinical trials by screening out clinical trials with poor expected results in the design stage.

[0038] Here, a clinical trial refers to an experimental method of randomized grouping of individuals as research subjects. Its purpose is to evaluate the efficacy of a new drug or new treatment for a certain disease, including whether it can prevent recurrence and prolong life.

[0039] When encountering a clinical problem, a clinician needs to translate the problem into a searchable and answerable question according to the “PICO” principle (the decomposed PICO is often used as the keyword for retrieval). PICO can be regarded as the abbreviation of patient P-intervention I-control C-outcome O, and its specific meaning is as follows:

[0040] P (patient / population): clinical characteristics of patients or people to be studied;

[0041] I (intervention or exposure): the treatment measure or exposure factor or treatment method of interest;

[0042] C (comparison): the control method, for example, "with placebo matched to the drug used"; and

[0043] O (outcome): the measurement index of interest.

[0044] The following shows examples of questions that can be asked according to the PICO principle.

[0045]

[0046] PICO provides a writing principle for medical literature. In existing medical literature, literature related to clinical trials is usually written in a manner consistent with the PICO principle as above. In addition, in the background and other parts of the literature, the background B (Background) of the clinical trial is usually included, thereby clarifying the situation faced by the clinical trial in design and facilitating the classification and search of the literature.

[0047] In the present application, the collected sentences containing comparison semantics are referred to as implicit evidence. "Implicit" evidence is a concept opposite to "explicit" evidence, which is evidence mined from conventional natural text, for example, unstructured and annotated literature is retrieved from a general literature database, and useful data hidden in natural text is found from it. In the present application, the useful data mined can be sentences containing comparison semantics as shown in the above example.

[0048] In the present application, the collected sentences containing comparison semantics are used as implicit evidence E ent including: collecting sentences containing comparison semantics from natural text as implicit evidence E ent for training. For example, implicit evidence collection can be performed from medical articles written in accordance with the PICO rule. These sentences are referred to as implicit evidence E ent , and also because they often implicitly contain the PICO components required by clinical evidence. For example, the sentence "artemether is more effective than artemisinin in inhibiting the proliferation of MCF-7 tumor cells", "MCF-7 tumor cells" are human breast cancer cells, so it implicitly indicates that the study population P is breast cancer patients. "Artemether" and "artemisinin" are related to the treatment method I and the control method C, respectively, and the measurement index O is related to "inhibition of the proliferation of MCF-7 tumor cells". By implicitly containing the PICO components required by clinical evidence, the implicit evidence E entThe model is fed in and uses the comparison results as labels (as will be detailed below, as "pseudo labels") to enable the model to learn the relationship between the PICO components and the clinical trial results through training, thereby facilitating the prediction of the clinical trial results as follows.

[0049] To collect evidence broadly, sentences containing comparison semantics can be collected from natural text contained in specific domain data sources as implicit clinical evidence E ent for training. For example, to train a clinical trial result prediction model, evidence can be collected from natural text contained in medical article databases. To train a fitness and diet behavior result prediction model, evidence can be collected from professional websites, fitness platforms, etc. For example, the present application can collect implicit clinical evidence broadly from existing medical literature. The largest general free medical database currently available is PubMed. In one embodiment, the PubMed (and PubMed Central) database can be used to collect comparison meaning sentences. Here, PubMed is a free search engine and a database containing approximately 30 million English biomedical article abstracts. PubMed Central is a database containing approximately 6 million English biomedical article full texts. The present application can extract tens of millions of pieces of implicit evidence from the above databases.

[0050] Due to the large size of the database literature, it is not feasible to manually find and label the evidence. Therefore, the present application can use a heuristic approach to collect large-scale unstructured implicit evidence for learning. Specifically, the above collection can be achieved by finding sentences containing predetermined comparison templates in the natural text as the implicit clinical evidence E ent . Extracting comparison results from the implicit clinical evidence includes extracting the comparison templates found in the implicit clinical evidence as comparison results r. Here, the comparison template can refer to an expression obtained through small-scale data analysis or an existing expression for expressing comparison between different objects, such as "stronger than" in the above example, or other fixed combinations such as "improve than", "decrease than", etc. In addition to expressions of rising and falling semantics, expressions of semantics similar to the compared objects such as "similar", "similar", "no change compared to" can also be used as comparison templates. The system can determine the comparison template to be used, and through template matching, automatically find sentences containing the above template as sentences containing comparison semantics, and process them to obtain comparison results r and comparison content E dis .

[0051] In the present application, the comparison result r extracted based on the matching with the comparison template can be regarded as data labels, more specifically, labeled data automatically obtained in a heuristic manner, different from manual labeling, which can also be referred to as "pseudo-labeled data".

[0052] Data is the basis of machine learning, which is built on data modeling, and rich labels are the premise of successful modeling of machine learning. Supervised learning is the most widely used machine learning algorithm at present, which strongly depends on labeled data. It constructs a prediction model by learning a large number of labeled training samples. Deep learning also needs a large amount of data "feeding", and machine learning frameworks represented by deep learning need to be trained on large supervised data sets. Data labeling is mainly for labeling speech, images, text, etc., mainly through marking, highlighting, tagging, object framing, and annotating to label data sets, and then training and learning these data sets by machines. The pseudo-labeled data (for example, comparison result r) used in the present application is also a kind of labeled data, which is automatically obtained in a heuristic manner, and is not manually labeled completely accurate data.

[0053] In addition to automatically obtaining pseudo-labeled data by using a heuristic method, the present application can also realize data enhancement by constructing reverse order evidence. For this purpose, the present application can also include: flipping the comparison content E dis and the comparison result r at the same time, thereby obtaining the reverse order comparison content E rev and the flipped comparison result Rev(r).

[0054] Here, the flipping of the comparison content can refer to the reversal of the original comparison object position (i.e., reverse order), and the flipping of the comparison result is the opposite of the result attribute. For example, for the sentence "artemether is stronger than artemisinin in inhibiting the proliferation of MCF-7 tumor cells", in the positive order construction, input "artemether is stronger than artemisinin in inhibiting the proliferation of MCF-7 tumor cells []", and train the model to input the correct comparison result r, that is, to fill in "strong" in the []. In the reverse order construction, the positions of the comparison objects "artemether" and "artemisinin" in the original sentence are reversed, and the opposite is "weak". Therefore, input E rev , that is, "artemisinin is weaker than artemether in inhibiting the proliferation of MCF-7 tumor cells []", and train the model to input the correct comparison result Rev(r), that is, to fill in "weak" in the [].

[0055] When the comparison result r itself is of similar semantics, the flipped comparison result Rev(r) can remain unchanged, and only the positions of the comparison objects are changed, and the same correctly expressed data can be obtained for model training. For example, based on the comparison template "no significant difference" or "no … difference", the sentence "There is no significant difference in the number of 1-year survival between the taxane-containing treatment and the control group after chemotherapy" is found as implicit evidence. The positive sequence sample can include the comparison content E dis "the number of 1-year survival between the taxane-containing treatment and the control group after chemotherapy", and its comparison result r "no significant difference", and the reverse sequence sample can include the comparison content E rev "the number of 1-year survival between the taxane-containing treatment and the control group after chemotherapy", and the comparison result Rev(r) "no significant difference", wherein the comparison content E dis and E rev is flipped, and the comparison result r and Rev(r) remain unchanged.

[0056] The data obtained after data augmentation by constructing reverse sequence samples can be used for model training. For this purpose, the method can further include: training the result prediction model according to the reverse sequence comparison content E rev and the flipped comparison result Rev(r). Alternatively, the above method can be incorporated into step S130, and for this purpose, step S130 can include: using the comparison content E dis and its comparison result r as a positive sequence sample, and using the flipped comparison content E rev and the flipped comparison result Rev(r) as a reverse sequence sample to train the clinical trial result prediction model. Adding reverse sequence examples helps the model to learn the comparison between the treatment group and the control group, rather than simply learning the co-occurrence relationship in the language model. For example, by adding reverse sequence samples, the model not only learns that "artemether-lumefantrine" and "artemisinin" and "strong" appear together in a sentence, but also truly learns the comparison relationship, that is, "artemether-lumefantrine" is "stronger" than "artemisinin", and "artemisinin" is "weaker" than "artemether-lumefantrine".

[0057] In addition, in order to better train the model, for example, to make the model learn the background in which the implicit evidence is established, the article content corresponding to the sentence containing the comparison semantics can also be collected as the background B of the implicit evidence. The article content used to collect the background B is usually specific article content, such as the background part of the article abstract. The background B is also used for training the clinical trial result prediction model, for example, is input in association with the corresponding positive sequence sample and / or reverse sequence sample.

[0058] In the present application, the data used for training the model can not be exactly the same as the data input and output when the model makes a prediction. Specifically, the present application can use sentences containing comparative semantics and their context B to construct training data. The model trained by the method of the present application can make behavior result prediction by inputting behavior data, for example, behavior data with a specific structure. For example, in the field of clinical trials, the BPICO (background B + PICO component) as described above can be used to predict the results of clinical trials.

[0059] The training of the present application as described above is a task of predicting the dug-out words of natural text, rather than a task of inputting BPICO and predicting R (Result). In other words, in the present application, the model "training" is different from the actual application task, and the input and output are also different, so in the field of machine learning, with reference to the final task, the training data used in the present application can be more accurately described as "pre-training data", and the task completed by training can be more accurately described as "pre-training task". For this reason, in the present application, training the result prediction model can include: pre-training the model for result prediction, such as a sequence modeling model. Specifically, the above-mentioned model can be a Transformer encoder whose model parameters are initialized by BioBERT.

[0060] The Transformer is a sequence modeling model, a deep learning model introduced in 2017, mainly used in the field of natural language processing (NLP).

[0061] Like a recurrent neural network (RNN), the Transformer is designed to process sequential data (such as natural language) to perform tasks such as translation and text summarization. However, unlike RNN, the Transformer does not need to process sequential data sequentially. For example, if the input data is a natural language sentence, the Transformer does not need to process the beginning of it before the end.

[0062] Since its inception, the Transformer has become the preferred model for solving many problems in NLP, replacing older RNN models such as long short-term memory (LSTM). Because the Transformer model facilitates more parallelization during training, it can train on larger datasets than before. This has led to the development of pre-training systems such as BERT and GPT, which have been trained on massive general-purpose language datasets and can be fine-tuned for specific language tasks.

[0063] BERT is a Transformer encoder pre-trained with a language model. BioBERT is a BERT pre-trained with biomedical text data. To this end, the present application can further pre-train the BioBERT pre-trained with biomedical text data with comparative semantic sentences, thereby obtaining the model for behavior outcome prediction of the present application. In this pre-training process, cross-entropy can be used as a loss function. To this end, step S130 can further comprise: the pre-training process optimizes the cross-entropy between the predicted result and the actual result.

[0064] The trained model for behavior outcome prediction can then be used to make behavior outcome prediction. Figure 2 A schematic flowchart of a method for outcome prediction according to an embodiment of the present application is shown.

[0065] In step S210, the behavior data to be performed is input into the outcome prediction model obtained as above. In step S220, the behavior outcome predicted by the outcome prediction model is obtained.

[0066] The data input into the outcome prediction model comprises the behavior data of the specific field, and the obtained behavior outcome is the prediction result of the target indicator being raised, lowered or unchanged. Further, the above behavior data can be data conforming to a specific format. For example, running three times a week, each time for five kilometers, with a pace of 6. Accordingly, in order to provide prediction accuracy, the context of the behavior to be performed can also be input into the outcome prediction model. For example, in the prediction of fitness behavior, the individual's physical condition, etc. can also be input as the context.

[0067] The above outcome prediction model can be a model for predicting a trial, in particular a clinical trial. To this end, Figure 2 The method shown can be implemented as a clinical trial outcome prediction method, comprising: inputting the data of the trial to be performed into the clinical trial outcome prediction model obtained as described above; and obtaining the clinical trial outcome predicted by the clinical trial outcome prediction model.

[0068] The data input into the outcome prediction model can comprise the PICO components of the trial data to be performed, and the obtained trial result R is the prediction result of the measurement indicator O being raised, lowered or unchanged. Further, the method can further comprise: inputting the research context B of the clinical trial to be performed into the outcome prediction model.

[0069] The pre-training method based on large-scale implicit evidence for implementing the clinical trial outcome prediction task will be described as follows Figure 3 and Figure 4 To implement the clinical trial outcome prediction task, a pre-training method based on large-scale implicit evidence is performed. To this end, Figure 3 A schematic diagram of the pre-training method based on large-scale implicit evidence is shown, Figure 4A schematic diagram of a result prediction task is shown. Figure 3 and Figure 4 One specific implementation example. Figure 1 and Figure 2 One specific implementation example.

[0070] To implement result prediction, for example, trial result prediction, as shown in Figure 3 and Figure 4 First, implicit evidence is collected using a heuristic method; then, a comparative language model is pre-trained using the implicit evidence; finally, the pre-trained model is used for clinical trial result prediction.

[0071] 1. Collecting implicit evidence

[0072] Clinical evidence is often expressed in a comparative form, such as “Remdesivir has better efficacy in treating COVID-19 than the control group”. For this purpose, the present application proposes to collect all sentences containing comparative semantics in medical literature databases, such as PubMed and PubMed Central, using a keyword matching method. Since PubMed and PubMed Central are English literature databases, matching needs to be performed against words, phrases or collocations containing comparative semantics in English. Here, the comparative semantics can include both the semantics of increase and decrease, and the semantics of similarity. To find sentences expressing the semantics of increase and decrease, sentences containing “than” are matched, and further matched with comparative degrees (e.g., comparative degrees of 32 adjectives or adverbs as comparative templates), such as “higher”, “smaller”, etc. And sentences containing both “than” and one or more comparative degrees are collected. To find sentences expressing the semantics of similarity, sentences containing “no difference between” and “similar to” patterns (e.g., using the above collocations as comparative templates representing the semantics of similarity) are matched. These sentences are referred to as implicit evidence E ent Because, compared to structured evidence, they often implicitly contain the PICO components required by clinical evidence. In order to enable the model to learn the background conditions of trial execution, the background part and the method part in the article abstract corresponding to these sentences are also collected as the background B of the implicit evidence.

[0073] This method can extract 11.8 million pieces of implicit evidence from PubMed and PubMed Central, of which 2.4 million pieces express a decrease in results, 3.5 million pieces express a similar result, and 5.9 million pieces express an increase in results.

[0074] 2. Pre-training comparative language model

[0075] After obtaining the implicit evidence, it is first processed to obtain samples for model training. The words in the collected implicit evidence that prompt the result semantics can be removed from E ent to obtain E dis and r, where r is the template corresponding to the word prompting the result, including 32 raise / lower templates and 2 similar templates.

[0076] The present scheme further flips E dis and r simultaneously for data augmentation, the principle of which is that when the comparison direction in the implicit evidence is opposite, the comparison result is also opposite to the original result. Denoted as reverse order implicit evidence E rev and the flipped comparison result Rev(r).

[0077] As shown in the figure, a sentence containing comparison semantics is matched from PubMed and PubMed Central using the comparison template containing "lower than", that is, "levels of viral antigen staining in lung sections of GS-5734-treated animals were significantly lower than vehicle-treated animals." Thus, the implicit data E ent is automatically mined according to the comparison template by the heuristic method. Subsequently, the words in the collected implicit evidence that prompt the result semantics can be removed from E ent to obtain E dis . As shown in the figure, "were significantly lower than" can be removed from the implicit evidence, and the remaining part obtains the comparison content "levels of viral antigen staining in lung sections of GS-5734-treated animals [MASK] vehicle-treated animals.", that is, "levels of viral antigen staining in lung sections of GS-5734-treated animals [MASK] vehicle-treated animals." and finds the comparison result r "lower than" from the result semantics, that is, "lower than". Thus, the positive order sample E dis and r are obtained. It should be understood that in the processing of implicit data, irrelevant words in the result semantics can be removed, for example, "were" in "were significantly lower than". In the processing of implicit data, the comparison content can be masked, for example, "levels of viral antigen staining in lung sections of GS-5734-treated animals [MASK] vehicle-treated animals." Figure 3In the illustrated embodiment, the word "significantly" is also removed, so that the trained model, when used for prediction, will output one of Figure 4 the three results: increase, no change, or decrease. In other embodiments, the comparison result r can also include a word indicating degree, such as "significantly" in the above example. Thus, the trained model, when used for prediction, will output one of Figure 4 more results: increase, no change, or decrease, such as "significantly increase", "slightly decrease", etc. However, the introduction of the degree word will greatly complicate the comparison template and increase the difficulty of model training. The model can be further pre-trained on the basis of the pre-trained model to obtain the training of the model indicating the degree of the result. Figure 4

[0078] After obtaining the positive order samples, the comparison content can be processed in reverse order to obtain the reverse order (or inverse order) samples. In Figure 3 the example, the comparison content E dis "levels of viral antigen staining in lung sections of GS-5734-treated animals [MASK] vehicle-treated animals." is reversed to "vehicle-treated animals [MASK] levels of viral antigen staining in lung sections of GS-5734-treated animals.", i.e., "solvent-treated animals [mask] GS-5734-treated animals of lung sections of viral antigen staining levels." The reverse of the comparison result r "lower than" is "higher than", i.e., "higher than". Thus, the flipped implicit evidence and comparison result are E rev and Rev(r) as the reverse order samples.

[0079] Further, the background of the literature where the implicit evidence is located can also be obtained, for example, "Emerging viral infections are difficult to control because heterogeneous members periodically [...]" shown in the figure, i.e., "emerging viral infections are difficult to control because of the periodicity of heterogeneous members [...]", and as the research background B, together with the implicit evidence obtained by the previous processing and the implicit evidence of the reverse order, input the Transformer model to train it to be able to target the comparison content E dis ​Output the comparison result r(“lower”), and output the inverted comparison content E. rev Output the comparison result Rev(r)(“higher”).

[0080] Therefore, this invention improves existing language models by pre-training a Transformer encoder model using a comparative language model to gain the ability to predict clinical trial outcomes. Specifically, two sets of samples are used for pre-training: 1) using positive-order implicit evidence E dis Predicting the outcome r; 2) Using implicit evidence E in reverse order rev Predict the opposite result Rev(r). Including reversed examples helps the model learn comparisons between the treatment and control groups, rather than co-occurrence relationships as in language models.

[0081] The pre-trained Transformer encoder model parameters are initialized by BioBERT, and the pre-training process optimizes the cross-entropy between the predicted and actual results. In other embodiments, other sequence modeling models, such as LSTM, CNN, etc., can also be pre-trained.

[0082] After training on massive amounts of data, such as millions of data points extracted from PubMed (e.g., a pre-trained Transformer encoder), a model is obtained for predicting clinical trial outcomes.

[0083] 3. Prediction of clinical trial results

[0084] Clinical trial outcome prediction tasks such as Figure 4 As shown. Specifically, the input to this task is in the form of natural text: 1) Clinical trial background B, such as “Given no specific antiviral therapy for COVID-19 [...]”; 2) The population to be studied P, such as “COVID-19 patients with severe [...]”; 3) Treatment method I, such as “Active emdesivir”; 4) Control method C, such as “Placebo matched remdesivir”; and 5) Measurement indicator O, such as “Time to Clinical Improvement”. The output is the result R, which is the comparison of the measurement result O between the treatment group I and the control group C in the study population P, which can be either increased, decreased, or unchanged.

[0085] The PICO elements of a new clinical trial to be studied are spliced into E, and E is input into the pre-trained input E dis and E rev Equivalently, E and its research background B are input into the pre-trained model described above, and the predicted comparison result r is output, and the 34 possible templates of r can be directly matched to the R of the result change direction, thereby predicting the result of the clinical trial.

[0086] The clinical trial result prediction model of the present scheme can accept any natural language input, so it can model a wider range of interventions. The present scheme collects large-scale unstructured implicit evidence in a heuristic way for learning, avoiding the cost of structured explicit evidence labeling. Further, the present scheme uses a comparative language model for pre-training, which can better learn the comparison between the treatment group and the control group, rather than the simple co-occurrence relationship in the language model.

[0087] Although Figure 3 and Figure 4 give examples of English natural text language, it should be understood that Chinese data sources (e.g., Chinese medical databases) can also be used to extract natural text comparison sentences, and Chinese result prediction models can be trained. The present invention does not limit the training and prediction languages.

[0088] In other embodiments, the present scheme can also be implemented as a result prediction model training device. Figure 5 The composition diagram of the result prediction model training device according to an embodiment of the present invention is shown. The training device 500 can perform the functions as shown in Figure 1 and Figure 3 , and train a result prediction model, for example, pre-train a transformer model.

[0089] As shown in the figure, the training device 500 includes a collection unit 510, an extraction unit 520, and a training unit 530.

[0090] The collection unit 510 is used to obtain (e.g., collect) sentences containing comparison semantics as implicit clinical evidence E ent for training. The extraction unit 520 is used to obtain comparison content E ent and comparison result r from the implicit evidence E dis , for example, extract the comparison result from the implicit clinical evidence, thereby obtaining the comparison content E dis and the comparison result r. The training unit 530 is used to train a result prediction model according to the comparison content E dis and the comparison result r, for example, train a clinical trial result prediction model using the comparison content E dis and its comparison result r.

[0091] In one embodiment, the collecting unit 510 can collect sentences containing comparison semantics from natural texts contained in a specific field data source, such as a medical article database, as implicit evidence E ent for training. ent The implicit evidence E

[0092] The collecting unit 510 can include a searching subunit configured to search for sentences containing a predetermined comparison template in the natural texts as the implicit evidence E ent wherein the comparison template hit by the searching is extracted from the implicit evidence as the comparison result r.

[0093] In one embodiment, data augmentation can be performed. Thus, the extracting unit 520 can include an inverse order constructing subunit configured to reverse the comparison content E dis and the comparison result r at the same time to obtain the inverse order comparison content E rev and the reversed comparison result Rev(r). The training unit 530 can be configured to train the result prediction model according to the inverse order comparison content E rev and the reversed comparison result Rev(r). Specifically, the training unit 530 can use the comparison content E dis and the comparison result r as a positive order sample, and use the reversed comparison content E rev and the reversed comparison result Rev(r) as an inverse order sample, to train the clinical trial result prediction model.

[0094] Further, the collecting unit 510 can be configured to collect article content (e.g., the background part in the abstract of article 0) corresponding to the sentences containing comparison semantics as the background B of the implicit evidence, which is used by the training unit for training of the clinical trial result prediction model.

[0095] In other embodiments, the scheme of the present application can also be implemented as a result prediction device. Figure 6 A composition diagram of a clinical trial result prediction device according to one embodiment of the present application is shown. As shown in the diagram, the prediction device 600 can include a data input unit 610 and a result obtaining unit 620. The data input unit 610 is configured to input data of a to-be-performed behavior into the result prediction model obtained as described above. The result obtaining unit 620 is configured to obtain a trial result predicted by the result prediction model.

[0096] In a preferred embodiment, the data input into the result prediction model can include PICO components of the to-be-performed trial data, and the obtained trial result R is a prediction result of whether a measurement index O is increased, decreased or unchanged.

[0097] In a preferred embodiment, the data input unit 610 can also be used to input the research background B to be tested into the result prediction model.

[0098] Figure 7 A structural diagram of a computing device according to an embodiment of the present application is shown, which can be used to implement the above-mentioned model training and result prediction.

[0099] Referring to Figure 7 The computing device 700 includes a memory 710 and a processor 720.

[0100] The processor 720 can be a multi-core processor, or can include multiple processors. In some embodiments, the processor 720 can include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processor (GPU), a digital signal processor (DSP), and the like. In some embodiments, the processor 720 can be implemented using a customized circuit, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0101] The memory 710 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 720 or other modules of the computer. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-write storage device or a volatile read-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 710 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 710 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and an instantaneous electronic signal transmitted by wireless or wired transmission.

[0102] The executable code stored on the memory 710 can cause the processor 720 to perform the model training and result prediction methods described above when the executable code is processed by the processor 720.

[0103] The model training and result prediction methods and devices according to the present application have been described in detail above with reference to the accompanying drawings. The present application can predict the behavior results by widely utilizing the existing behavior effects by training the result prediction model by aggregating the massive comparative information in natural text, thereby reducing the trial and error cost.

[0104] As described in detail above, in order to assist the design of clinical experiments, the present scheme predicts the results of clinical experiments by a large-scale implicit evidence pre-training method. In the present scheme, the clinical trial result prediction model can accept any natural language input, so it can model a wider range of interventional treatments. The present scheme collects a large amount of unstructured implicit evidence in a heuristic way for learning, avoiding the cost brought by structured explicit evidence annotation. Further, the present scheme is pre-trained by a comparative language model, which can better learn the comparison between the treatment group and the control group, rather than the simple co-occurrence relationship in the language model. The scheme can obtain significant result improvement on the standard data set.

[0105] For the prediction of results in other fields, a similar method can be used to extract comparative sentences from natural text in different data sources for training. For example, in the fitness field, comparative sentences for various fitness behaviors can be widely scraped from professional websites or fitness platforms and other data sources. Here, fitness behaviors can include various sports activities, as well as related behaviors such as diet control, sleep improvement, and physiotherapy massage. The predicted results can be the changes in one or more fitness indicators, such as body weight, body fat rate, and waist circumference. In addition, comparative sentences for improving (or just changing) mood can be extracted from, for example, more extensive data sources to train a mood change result prediction model. Here, mood-related behaviors can include various behaviors, such as work and academic stress, exercise and fitness habits, season, and the presence or absence of bad habits. The predicted results can be subjective evaluations of mood changes, such as mood measurement cards designed for psychology.

[0106] In addition, the method according to the present application can also be implemented as a computer program or computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present application.

[0107] Alternatively, the present application can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having stored executable code (or computer program, or computer instruction code) which, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the above method according to the present application.

[0108] Those skilled in the art will also appreciate that the various example logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of the two.

[0109] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0110] Embodiments of the application have been described above with the understanding that such embodiments are exemplary, but are not exhaustive, and are not limited to the embodiments disclosed. Many modifications and variations of the described embodiments are possible in light of the above teachings. The choice of terms to be used in the description is intended to best describe the principles of the embodiments, practical application, or improvement over the technology in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a result prediction model, comprising: Obtaining sentences containing comparative semantics as implicit evidence E for training ent ; From the implicit evidence E ent comparative content E dis and the comparison result r ; According to the comparison content E dis and the comparison result r , the result prediction model is trained; wherein obtaining sentences containing comparative semantics as implicit evidence E for training ent comprising collecting sentences containing comparative semantics from natural text as implicit evidence E for training ent ; wherein the method further comprises: obtaining reverse comparison content Erev and reversed comparison result Rev(r) according to the comparison content Edis and the comparison result r; and training the result prediction model according to the reverse comparison content Erev and the reversed comparison result Rev(r).

2. The method of claim 1, wherein, collecting sentences containing comparative semantics from natural text as implicit evidence E for training ent comprising: collecting sentences containing comparative semantics from natural text contained in a specific domain data source as implicit evidence E for training a specific domain outcome prediction model ent .

3. The method of claim 1, wherein, Collecting sentences containing comparative semantics from natural text as implicit evidence E for training ent comprising: finding a sentence in the natural text containing a predetermined comparison template as the implicit evidence E ent , obtaining a comparison result from the implicit evidence E ent r comprising:​ extracting from the implicit evidence E ent the comparison template that the aforementioned search hits, as the comparison result r .

4. The method of claim 1, wherein, Training the result prediction model comprises: Pre-training a sequence modeling model for result prediction.

5. The method of claim 4, wherein, Pre-training the sequence modeling model for result prediction comprises: The pre-trained model parameters are initialized by a Transformer encoder of BioBERT.

6. The method of claim 4, wherein, Pre-training the sequence modeling model for result prediction comprises: The pre-training process optimizes the cross-entropy between the predicted result and the actual result. 7.The method of claim 1, further comprising: Obtaining article content corresponding to the sentence containing comparison semantics as background B of the implicit evidence, which is also used to train the result prediction model. 8.A result prediction method, comprising: Inputting behavior data to be performed into the result prediction model obtained in any one of claims 1-7; and Obtaining a behavior result predicted by the result prediction model.

9. The method of claim 8, wherein, The data input into the result prediction model comprises the behavior data of the specific field, and the obtained behavior result is a prediction result of target indicator rising, falling or being unchanged. 10.The method of claim 9, further comprising: Inputting background of the behavior to be performed into the result prediction model. 11.A clinical trial result prediction method, comprising: Inputting data of a trial to be performed into the result prediction model obtained in any one of claims 1-7, wherein the result prediction model is a clinical trial result prediction model; and Obtaining a clinical trial result predicted by the clinical trial result prediction model.

12. The method of claim 11, wherein, The data input into the clinical trial result prediction model comprises PICO components of the data of the trial to be performed, and the obtained trial result is a prediction result of measurement indicator O rising, falling or being unchanged, wherein the PICO is patient P-intervention I-contrast C-measurement indicator O. 13.The method of claim 12, further comprising: Inputting research background B of the trial to be performed into the clinical trial result prediction model. 14.A training device for a result prediction model, comprising: a collecting unit for obtaining a sentence containing a comparative semantics as implicit evidence E for training ent ; an extraction unit for obtaining comparison content E ent from the implicit evidence E dis and a comparison result r ; and a training unit configured to train a result prediction model based on the comparison content E dis and the comparison result r the training result prediction model; The collection unit is further configured to collect sentences containing comparative semantics from natural text as implicit evidence E for training ent ; wherein the extraction unit comprises a reverse construction subunit configured to reverse the comparison content Edis and the comparison result r simultaneously to obtain reverse comparison content Erev and reversed comparison result Rev(r), and the training unit is configured to train the result prediction model according to the reverse comparison content Erev and the reversed comparison result Rev(r).

15. The apparatus of claim 14, wherein, The collection unit comprises: A searching subunit is configured to search for a sentence containing a predetermined comparison template in the natural text as the implicit evidence E ent wherein the comparison template hit by the searching is extracted from the implicit evidence as the comparison result r .

16. The apparatus of claim 14, wherein, The collection unit is configured to: Collect article content corresponding to the sentence containing comparison semantics as background B of the implicit evidence, which is used by the training unit to train the result prediction model. 17.A result prediction device, comprising: a data input unit, configured to input behavior data to be performed into the result prediction model obtained in any one of claims 1-7; and a result obtaining unit, configured to obtain a behavior result predicted by the result prediction model. 18.A computing device, comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method of any one of claims 1-13. 19.A non-transitory machine-readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-13.

Images