Adversarial training-based rct literature classification model training method and system

Abstract

The application provides an RCT literature classification model training method and system based on adversarial training, the method comprising: obtaining a specified medical literature dataset with annotations; wherein the annotations include RCT literature, non-RCT literature, and divergent literature; the divergent literature refers to literature whose judgment of whether it belongs to RCT literature is different among multiple annotators; constructing an adversarial model based on the literature annotated as divergent literature in the dataset, and training the adversarial model based on the literature annotated as RCT literature and non-RCT literature in the dataset; running the adversarial model, generating a sample library, and training a binary classification model based on the sample library; wherein the binary classification model is a machine learning model for identifying whether literature belongs to RCT literature, and the proportion of RCT literature and non-RCT literature in the sample library meets a preset sample balance condition. The application improves the literature classification accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of document classification, and particularly relates to an RCT document classification model training method and system based on adversarial training. BACKGROUND

[0002] With the development of medical concepts, the current medical model has changed from the past experience medicine to evidence-based medicine (EBM). Evidence-based medicine adheres to the principle that all clinical decisions should be based on clinical evidence, which can provide the strongest evidence support and rigorous clinical research design guidance for medical clinical work, and has important guiding significance for clinical practice and research. The main evidence carrier of evidence-based medicine is systematic review, which has extremely strict writing requirements. Researchers need to find the current best clinical evidence by conducting systematic retrieval and literature screening for a specific clinical problem, and evaluate the bias risk of these evidence and integrate the results. Its steps involve systematic retrieval, literature screening, information extraction, bias risk evaluation and data synthesis. In order to control the bias risk of the included literature itself, the current best clinical evidence that the systematic review writer needs to find is generally the most rigorous randomized controlled clinical trial (RCT) in research design. In order to ensure the systematicness of retrieval, the evaluator often uses a retrieval strategy with high recall rate (sensitivity) but low precision rate (accuracy), so that the literature screening link often faces thousands of title abstracts, and most of the abstracts are not RCTs, which will consume a lot of time and effort of researchers, and at the same time, the writing and updating speed of systematic review cannot meet the needs of clinical decision-making. A study shows that it takes an average of 1781 papers to evaluate a systematic review, and the average screening rate of irrelevant literature is 97.1%, and it takes an average of 64.3 weeks to publish a systematic review.

[0003] Although the literature screening itself requires strictness, complex process and large workload, the intelligent classifier currently developed for this purpose still has the following defects:

[0004] 1. Data annotation defects;

[0005] 2. Annotation noise: the Kappa coefficient of traditional manual annotation is only 0.65 (ideal value > 0.9);

[0006] 3. Sample imbalance: lack of diversity of non-RCT literature (the existing data set only covers 42 types of research);

[0007] 4. The recall rate still needs to be further improved and optimized;

[0008] 5、The existing RCT screening literature recall scheme can further reduce the number of non-RCT literature incorrectly labeled as RCT (high false positive rate), but at the same time, the number of non-RCT literature incorrectly labeled as RCT (false positive rate) is too large. SUMMARY

[0009] The embodiment of the present application provides an RCT literature classification model training method and system based on adversarial training, which can solve the problem of insufficient RCT literature classification accuracy.

[0010] In a first aspect, the embodiment of the present application provides an RCT literature classification model training method based on adversarial training, comprising the following steps:

[0011] Obtain a specified medical literature dataset with annotations; wherein the annotations include RCT literature, non-RCT literature and divergent literature; the divergent literature refers to literature whose judgment of whether it belongs to RCT literature is different among multiple annotators;

[0012] Construct an adversarial model based on the literature in the dataset that is annotated as divergent literature, and train the adversarial model based on the literature in the dataset that is annotated as RCT literature and non-RCT literature;

[0013] Run the adversarial model to generate a sample library, and train a binary classification model based on the sample library; wherein the binary classification model is a machine learning model for identifying whether the literature belongs to RCT literature, and the proportion of RCT literature and non-RCT literature in the sample library meets a preset sample balance condition.

[0014] In a possible implementation manner of the first aspect, the binary classification model includes a semantic understanding model and a fully connected prediction layer; wherein the semantic understanding model is used to extract features; and the fully connected prediction layer is used to predict the probability of the literature belonging to RCT literature according to the features extracted by the semantic understanding model, and output a binary classification result according to the probability.

[0015] In a possible implementation manner of the first aspect, the semantic understanding model includes:

[0016] A first feature extraction layer is used to extract the research framework features of the literature.

[0017] In a possible implementation manner of the first aspect, the semantic understanding model includes:

[0018] A second feature extraction layer is used to extract the hidden exclusion features of the literature.

[0019] In a possible implementation manner of the first aspect, the outputting of the binary classification result according to the probability includes:

[0020] determining that the probability is greater than a threshold value, outputting a binary classification result that the literature belongs to RCT literature, otherwise outputting a binary classification result that the literature belongs to non-RCT literature;

[0021] In a possible implementation of the first aspect, the threshold value is a dynamic threshold value adjusted according to a literature publication year.

[0022] Compared with the prior art, the embodiments of the present application have the following advantages:

[0023] 1. An adversarial model is constructed using literature with divergent annotations, so as to obtain a better sample expansion model, and more effective sample data can be obtained based on the model to improve the training effect of the binary classification model.

[0024] 2. The sample is expanded using the adversarial model, and the binary classification model is selectively trained using a balanced sample ratio, so that a binary classification model with higher classification accuracy (i.e., better accuracy, precision, and recall) can be obtained.

[0025] In a second aspect, the embodiments of the present application provide an RCT literature classification model training system based on adversarial training, comprising:

[0026] A data set module is configured to obtain a specified medical literature data set with annotations. The annotations include RCT literature, non-RCT literature, and divergent literature. The divergent literature refers to literature for which multiple annotators have different judgments as to whether the literature belongs to RCT literature.

[0027] An adversarial module is configured to construct an adversarial model based on literature in the data set that is annotated as divergent literature, and train the adversarial model based on literature in the data set that is annotated as RCT literature and non-RCT literature.

[0028] A training module is configured to run the adversarial model, generate a sample library, and train a binary classification model based on the sample library. The binary classification model is a machine learning model for identifying whether literature belongs to RCT literature, and the proportion of RCT literature and non-RCT literature in the sample library meets a preset sample balance condition.

[0029] In a third aspect, the embodiments of the present application provide a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for training an RCT literature classification model based on adversarial training according to any one of the first aspect is implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the RCT document classification model training method based on adversarial training described in any one of the first aspects above.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a terminal device, enables the terminal device to execute the RCT document classification model training method based on adversarial training as described in any one of the above-mentioned first aspects.

[0032] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a comparison chart of the literature classification results of the binary classification model Diting Model of this embodiment and the similar small model Robotsearch;

[0034] Figure 2 This is a framework diagram of the RCT literature classification model training system based on adversarial training;

[0035] Figure 3 This is a frame diagram of the terminal device;

[0036] Figure 4 This is a specific flow chart of the actual operation of step 102. DETAILED DESCRIPTION

[0037] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0038] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0039] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0040] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0041] In the present application, the reference to "one embodiment" or "some embodiments" means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in the present description are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0042] The method and system for training an RCT literature classification model based on adversarial training will be described in more detail below in conjunction with the schematic drawings, in which the preferred embodiments of the present application are represented, it being understood that the person skilled in the art can modify the present application described herein while still achieving the advantageous effects of the present application.

[0043] The present application provides a method for training an RCT literature classification model based on adversarial training, comprising the following steps:

[0044] Step 102, obtaining a specified medical literature dataset with annotations; wherein the annotations include RCT literature, non-RCT literature and divergent literature; the divergent literature refers to literature whose judgment of whether it belongs to RCT literature is different among multiple annotators;

[0045] Step 104, constructing an adversarial model based on the literature in the dataset annotated as divergent literature, and training the adversarial model based on the literature in the dataset annotated as RCT literature and non-RCT literature;

[0046] Step 106, running the adversarial model to generate a sample library, and training a binary classification model based on the sample library; wherein the binary classification model is a machine learning model for identifying whether the literature belongs to RCT literature, and the proportion of RCT literature and non-RCT literature in the sample library meets a preset sample balance condition.

[0047] Regarding step 102:

[0048] In an optional embodiment, the medical literature dataset is a dataset obtained by manual annotation based on at least 2 annotators.

[0049] In a preferred embodiment, the medical literature dataset is based on expert annotation in the relevant field. Traditional literature annotation work is usually carried out by specialized annotators, and these annotators have varying levels of understanding of specific fields, which may result in a certain percentage of errors in the annotation results.

[0050] However, this annotation method is still widely used because the data demand involved in the annotation work is usually large; the number of experts in a specific field, such as medical experts in the medical field involved in the present application, is limited and the output per unit time is lower, and the cost of using experts for annotation is too high. Once the aforementioned problem of large data demand is added, the cost problem will be extremely prominent.

[0051] Meanwhile, the construction of the adversarial model in the subsequent scheme of the present embodiment depends on the conflictiveness of the annotation results, which can effectively improve the performance of the adversarial model, provided that the quality of the annotation results is high enough, that is, the scheme based on the idea of the present embodiment needs experienced annotators as much as possible, ideally experts in the medical field.

[0052] The specific process example of the actual operation of step 102 will be provided below, as shown in Figure 4

[0053] Based on the related research project currently implemented by the applicant, three experts constructed an initial seed library based on the Retraction Watch Database of retracted literature, annotated a total of 12,472 pieces of literature, classified the literature as a random control trial (RCT) by a consensus method, divided into an included group (Include, i.e., annotated as RCT literature) and an excluded group (Exclude, i.e., annotated as non-RCT literature), marked the MAYBE group (1 person and the other 2 people have opposite views or cannot make a judgment) with a larger difference, and finally marked 1,330 pieces of included literature and 11,142 pieces of excluded literature.

[0054] In the above example, the annotation results with differences are still finally marked as RCT or non-RCT, and in some other optional embodiments, the subsequent annotation of the difference group can also not be performed.

[0055] In addition, the number of experts participating in the annotation work is preferably 3, which can reduce the labor cost while ensuring the simplicity of subsequent annotation of the difference group (i.e., 2 people think it is the first result, and 1 person thinks it is the second result. The annotation can be performed by referring to the common views of 2 people), thereby providing higher quality samples for the training of the subsequent adversarial model.

[0056] ​Step 104 is used to expand the training samples of the binary classification model. The reason for selecting the adversarial network to expand the samples in this embodiment is that the typical RCT literature and the typical non-RCT literature usually have clear characteristics. According to the experimental results of the applicant, the typical RCT literature and the typical non-RCT literature rarely become the source of errors of the binary classification model. Therefore, the identification of the ambiguous samples (the MAYBE group) becomes a key factor for limiting the performance of the binary classification model.

[0057] The original intention of sample expansion is to overcome the problem of sample imbalance, that is, to make the proportion of RCT literature and non-RCT literature in the sample library meet the sample balance condition. Since the ambiguous samples have a more critical role in the performance of the binary classification model, step 104 of this embodiment also considers the performance tuning effect of the expanded samples on the subsequent binary classification model, that is, to try to generate more expanded samples similar to the ambiguous samples.

[0058] This beneficial effect is achieved through the construction idea of the adversarial model, that is, based on the ambiguous literature to construct the adversarial model.

[0059] The ambiguous literature can be regarded as a "blueprint of ambiguity". These literature are not simply noise data, but a set of samples with ambiguity identified by human annotators. This ambiguity is not randomly generated, but is derived from specific text features. An adversarial model "constructed" based on such literature can learn to replicate or target these specific ambiguity patterns, rather than simply applying random perturbations. This is a more complex form of adversarial attack / generation. If the adversarial model is a generator, it can be trained to generate new samples that mimic this blueprint; if it is a model that learns the ambiguity patterns, it can learn to identify the features of this blueprint.

[0060] Unconventional definition of adversarial model: the adversarial model of the present application is essentially different from the traditional generative adversarial network (GAN), and its purpose is not to generate false samples to deceive the discriminator, but to extract key ambiguity features (such as semantic ambiguity of research method description and incompleteness of randomization logic) that cause annotators to be ambiguous through deep analysis of ambiguous literature, and encode these features as the core discrimination rule of the adversarial model.

[0061] Directional feature mining of MAYBE group literature: 1. Use NLP semantic parsing technology to structurally decompose the MAYBE group literature, identify high-frequency ambiguity sentence patterns (such as the implied non-randomness of "according to the patient's wishes allocation") and contradictory text structures (such as mixing prospective design and retrospective analysis terms in the abstract); 2. When designing the loss function of the adversarial model, introduce the annotator disagreement index (such as the Kappa coefficient variance), and force the model to pay attention to the features of the MAYBE group literature during the training process, so that it can quantify the uncertain area of the classification boundary.

[0062] The following will provide a specific process example of step 104 actual operation of the applicant:

[0063] Since the typical inclusion and exclusion group literature text features are obvious, it is difficult to improve the recall rate and discrimination accuracy of the small model, therefore, the MAYBE group, that is, the literature entries with semantic ambiguity leading to expert disagreement, is used to construct a text adversarial system, the core of which is to use the "fuzziness" and "confusion" features contained in these divergent literature itself to guide the adversarial network (especially the generator) to learn and generate higher quality and more challenging adversarial samples. The results of high confidence labeling are divided into two groups, Include and Exclude, to construct an initial literature seed library for training;

[0064] For the text adversarial generator in the adversarial model:

[0065] For RCT literature: randomly insert non-randomization description sentences (such as "subjectively assigned by researchers");

[0066] For non-RCT literature: generate false positive paragraphs that meet the CONSORT specification.

[0067] When constructing the adversarial model, by analyzing these divergent literature, it can learn:

[0068] Which words, phrase combinations or sentence structures are likely to cause ambiguity?

[0069] What is not clear in the research method description that will cause the RCT and non-RCT boundary to be ambiguous?

[0070] How do non-RCT literature mimic the expression of RCT to achieve the effect of false appearance?

[0071] For step 106, in a preferred embodiment, the sample balance condition is that the number of RCT literature and non-RCT literature is comparable, and "comparable" here allows a certain percentage / amount of error.

[0072] According to the above embodiment, in yet another embodiment:

[0073] The binary classification model includes a semantic understanding model and a fully connected prediction layer; wherein the semantic understanding model is used to extract features; the fully connected prediction layer is used to predict the probability that the literature belongs to RCT literature according to the features extracted by the semantic understanding model, and output a binary classification result according to the probability.

[0074] In a preferred embodiment, the semantic understanding model is a machine learning model based on the BERT model.

[0075] BERT (Bidirectional Encoder Representations from Transformers) as a pre-training language model, its bidirectional attention mechanism can capture the long-distance context dependence in the text, which is particularly important for deep semantic analysis of medical literature. For example, when determining whether a paper belongs to a randomized controlled trial (RCT), the model needs to understand the relevance of multiple parts such as study design, methodology description, and result analysis. Traditional unidirectional models such as LSTM often have information decay problems when dealing with such cross-paragraph semantic associations, while BERT can dynamically adjust the weight of different positions through the multi-head attention layer in the self-attention mechanism.

[0076] In addition to BERT, there are many alternative model architectures that can be used for semantic understanding, each with unique advantages and limitations. For example, RoBERTa, as an improved version of BERT, performs better than the original BERT in multiple natural language understanding tasks through dynamic masking mechanisms and larger-scale training data. For the RCT literature classification task, RoBERTa may be better at capturing subtle semantic differences.

[0077] Another model worth noting is Longformer, which can handle long documents up to 4096 tokens by introducing a combination of local attention mechanisms and global attention mechanisms, which has unique value for scenarios that require complete analysis of entire medical literature.

[0078] In terms of lightweight models, ALBERT significantly reduces the number of model parameters through parameter sharing and embedding decomposition techniques, making it suitable for deployment in resource-constrained environments. For example, in online services that need to process a large number of literature streams in real time, ALBERT can achieve fast reasoning while maintaining reasonable accuracy. However, its performance ceiling is usually lower than that of the full version of BERT, especially when dealing with complex semantic reasoning tasks.

[0079] The ELECTRA model uses a replacement detection pre-training task, which is more efficient and more fully utilizes computing resources than BERT's masked language modeling. This is useful for scenarios that require frequent model updates (such as tracking the evolution of medical research methods), as it allows faster fine-tuning of new data. However, ELECTRA may be slightly inferior to traditional BERT architecture in capturing deep semantic relationships, which may have a slight impact on tasks that require a detailed understanding of study design.

[0080] In summary, the core task of semantic understanding models is feature extraction, and a significant portion of natural language understanding models can complete this task, but as mentioned above, BERT-like models often have certain advantages when facing the tasks in this application.

[0081] Of course, some BERT-based models also have shortcomings, for example, the limitation of input TOKEN makes the literature content cannot be fully understood, which leads to the extracted features are not complete and comprehensive.

[0082] In order to overcome these problems, one optional idea is to combine models, for example, use BERT-based models to extract features of key paragraphs of literature (such as title, abstract, etc.), use Longformer to complete the feature extraction of the full text of the literature, and set the weights of the two.

[0083] Therefore, the combined model can realize the extraction of different features of the same literature.

[0084] Continuing this idea, the present embodiment provides an optional implementation as follows.

[0085] The semantic understanding model comprises:

[0086] A first feature extraction layer for extracting research framework features of the literature.

[0087] A second feature extraction layer for extracting hidden exclusion features of the literature.

[0088] Wherein, the research framework features refer to the research idea logical chain in the literature, for example, the logical chain of “random allocation, double-blind, multi-center”.

[0089] The hidden exclusion features refer to some keywords in the text of some RCT literature, which can individually deny the RCT features of the literature. The existence of such keywords can to some extent identify the literature as non-RCT literature, for example, “retrospective cohort” mixed in the abstract.

[0090] This implementation can theoretically more effectively complete the identification of RCT literature, but since the binary classification model has poor interpretability for humans, it is difficult for us to find problems and fine-tune during the training process, therefore, on the basis of the above two feature extraction layers, we provide an optimal implementation as follows:

[0091] The semantic understanding model is constructed as a double-branch model.

[0092] Wherein, the first branch is as described above, for extracting research framework features and / or hidden exclusion features of the literature.

[0093] The second branch is used to output explanatory text of the research framework features and / or hidden exclusion features.

[0094] That is, on the basis of the original semantic understanding model output parameters, these parameters are "translated" into natural language, so that during the training process, researchers can count the "logical chain" in the research framework feature and / or the "keyword" in the hidden exclusion feature to estimate the effect of the training, and then fine-tune the training process to achieve better model performance.

[0095] According to the above embodiment, in yet another embodiment:

[0096] The outputting of the binary classification result according to the probability comprises:

[0097] If it is determined that the probability is greater than a threshold value, a binary classification result that the literature belongs to RCT literature is output, otherwise a binary classification result that the literature belongs to non-RCT literature is output.

[0098] Preferably, the threshold value is a dynamic threshold value adjusted according to the literature publication year.

[0099] It is particularly worth mentioning that the embodiments of the present application involve an adversarial model, a binary classification model and the labeling / extension of samples, because these embodiment schemes are used to explain the construction and training process of the binary classification model. When the binary classification model training work is completed, the model used to perform the RCT literature classification task is the trained binary classification model itself and does not involve others. That is:

[0100] In the training phase:

[0101] T1, it is necessary to obtain a labeled data set (step 102), and the labeling work of these data sets is completed by artificial—especially by experts in the relevant field—can effectively improve the reliability of the data (which constitutes a preferred pre-step of step 102), and then improve the training effect of the model to obtain a binary classification model with better performance (so as to have better performance when performing the RCT literature classification task).

[0102] T2, it is necessary to construct an adversarial model based on the divergent literature labeled as MAYBE in T1, and the construction here can also be understood as a kind of training for the adversarial model. After the training of the adversarial model is completed, it is necessary to use the trained adversarial model to expand the samples.

[0103] T3, the aforementioned samples are used to train the binary classification model, the sample library can include part of the samples output by the trained adversarial model, part of the samples in the data set obtained in step 102 and labeled as MAYBE, and part of the samples not obtained in step 102 (for example, the preferred solution of step 102 is to use a higher-quality data set labeled by medical professionals to optimize the performance of the adversarial model, and when the samples generated by the adversarial model are good enough, some samples labeled by ordinary annotators can also achieve good training effect of the binary classification model).

[0104] More specifically, in an optional embodiment, the samples in the sample library (i.e., the sample set used to train the binary classification model) are all labeled (i.e., the true value of whether the sample belongs to RCT or non-RCT), so that supervised training can be realized for the binary classification model.

[0105] In addition, considering the cost: the marginal cost of the samples generated based on the trained adversarial model is the lowest (i.e., excluding the data labeling cost for training the adversarial model, the cost of the samples generated by the adversarial model can be ignored), the marginal cost of the samples labeled by ordinary annotators is moderate (each piece of data needs to be labeled by working hours, but the hourly wage is relatively low), and the marginal cost of the samples labeled by experts in the relevant field (the preferred embodiment of step 102) is the highest, but due to the training needs of the adversarial model, this part of high-cost samples is essential.

[0106] Therefore, the preferred solution under the premise of considering the cost is to label enough data for the adversarial model training by experts, wherein the MAYBE group is used for training the adversarial model, and the RCT and non-RCT groups are used for training the binary classification model.

[0107] Subsequently, since the number of data in the RCT and non-RCT groups in the expert-labeled data cannot be estimated and determined (even if the data labeled by ordinary annotators is considered, the same problem exists), it may lead to the problem of unbalanced samples for training the binary classification model—such as the ratio of positive samples (labeled as RCT) and negative samples (labeled as non-RCT) is disproportionate; at the same time, such labeled data as samples may have the problem of too explicit binary classification characteristics (refer to the description above), which is not conducive to optimizing the training effect of the binary classification model.

[0108] Therefore, thanks to the ambiguous characteristics and almost zero marginal cost of the adversarial model generated samples, the above two problems can be better solved—i.e., generate enough and balanced samples by the trained adversarial model to make the samples in the sample library meet the requirements (preset sample balance conditions).

[0109] T4, the sample library based on T3 is used to train a binary classification model, and after the trained binary classification model is obtained, the training phase is completed, and the RCT literature classification task can be performed subsequently.

[0110] Unlike the training phase described above, the RCT literature classification task execution phase involves relatively simple steps, specifically:

[0111] E1, obtaining the literature to be classified.

[0112] Please note that the literature obtained in step E1 is the original literature without any labeling. Of course, the original literature can be datafied through some encoding, partial extraction, standardization and other preprocessing steps, but these preprocessing steps do not involve labeling personnel or labeling tags, true values.

[0113] E2, inputting the literature to be classified into the trained binary classification model.

[0114] E3, obtaining the literature labeling result output by the binary classification model, and the RCT literature classification task is completed.

[0115] Referring to the foregoing description, the following will be based on the actual implementation process of the applicant to provide relevant data to reflect the improvement effect of the present application.

[0116] The data of the related process is as follows.

[0117] The amount of training data: 1000 initial seeds (i.e. the amount of data labeled by experts in the field), expanded to 50,000 through adversarial expansion (including the amount of sample library data used to train the binary classification model generated by the trained adversarial model).

[0118] After training the binary classification model based on the 50,000 data, the performance of the trained binary classification model in executing the RCT literature classification task is compared with the performance of the traditional model in executing the RCT literature classification task as shown in Table 1.

[0119] Table 1

[0120] Test Set Traditional Model (F1) This Proposal (F1) Withdrawn RCTs 0.79 0.93 Child RCTs 0.68 0.89 Cohort 0.52 0.81

[0121] As can be seen, the model based on the above scheme has achieved better performance than the traditional model.

[0122] However, the aforementioned cost problem still has room for optimization.

[0123] A considerable portion of the literature not belonging to the MAYBE group "wastes" the ability of the experts in some sense (these literatures not belonging to the MAYBE group can be used as training samples for the binary classification model, but the output of the adversarial model can be replaced at a lower cost), if the proportion of RCT literatures / non-RCT literatures with clear characteristics in the data labeled by experts is reduced, and the proportion of MAYBE group literatures with ambiguous characteristics is increased, and the professional knowledge of experts is more fully utilized to exclude these data with clear characteristics of RCT literatures / non-RCT literatures, the expert group will complete more MAYBE group literature screening at a shorter time cost.

[0124] Then, the optimization goal of the cost problem is transformed into how to make the data set labeled by experts have a smaller proportion of RCT literatures / non-RCT literatures with clear characteristics and a larger proportion of MAYBE group literatures with ambiguous characteristics.

[0125] Considering that the preferred solution is that the better (trained) binary classification model itself includes a semantic understanding module, that is, the aforementioned solution can still achieve an observable performance improvement based at least in part on the model's understanding of semantics, which to some extent proves that machine learning models with semantic understanding capabilities can provide positive value in RCT literature identification.

[0126] Therefore, the cost optimization problem is further specified as how to use machine learning models to identify RCT literatures at a low cost.

[0127] Naturally, some open-source, pre-trained large language models (such as DeepSeek) are considered for performing the above task (a prerequisite step for step 102, labeling the data set provided to the experts with a smaller proportion of RCT literatures / non-RCT literatures with clear characteristics and a larger proportion of MAYBE group literatures with ambiguous characteristics).

[0128] Naturally, we continue this line of thought and add a rough screening step based on an open-source, pre-trained large language model (such as DeepSeek) before the trained binary classification model performs the RCT literature identification task. Unlike the final classification result, the preliminary classification result of the rough screening step is labeled as LLM-RCT and LLM-NonRCT.

[0129] Based on this optimized process, we provide Figure 1 the preferred embodiment shown in the figure, wherein:

[0130] For the part of the rough screening result of LLM-RCT, due to the aforementioned positive value, it is considered that the probability of these literatures belonging to RCT is higher, and therefore a secondary verification is carried out by using a different scheme from the other scheme of the application, which will not be described here.

[0131] For the part of the rough screening result of LLM-NonRCT, the aforementioned embodiments of the application can achieve quite ideal effects, Figure 1 The provided data can intuitively compare the performance of the DiTing Model (the code of the binary classification model after the training of the application) and the Robotsearch (as a representative of the traditional model). In Figure 1 the comparative example, the DeepSeek model performs rough screening on 8325 literatures (the true value of these literatures is that 780 belong to RCT literatures, Figure 1 marked as 780Include, and 7545 belong to non-RCT literatures, Figure 1 marked as 7545Exclude). In the result, there are 6695 literatures marked as LLM-NonRCT (the true value of these literatures is that 16 belong to RCT literatures, and 6979 belong to non-RCT literatures), and 1330 are marked as LLM-RCT (the true value of these literatures is that 764 belong to RCT literatures, Figure 1 marked as 764Include, and 566 belong to non-RCT literatures, Figure 1 marked as 566Exclude).

[0132] The two models both take the 6695 literatures with the rough screening result of LLM-NonRCT as input to perform comparison test, as shown in Figure 1 from the test result, although the DiTing Model (the code of the binary classification model after the training of the application) missed 8 Include literatures, which is slightly higher than 6 of the Robotsearch (as a representative of the traditional model), but the amount of literatures incorrectly marked as included is much less than that of the latter (130<1436).

[0133] The false negative rate of the system built by the Diting Model is 1.03%, and the false positive rate is 9.23%, the false negative rate of the system built by the Robotsearch is 0.77%, and the false positive rate is 26.53%, and the overall performance is greatly improved.

[0134] The application embodiment also provides an RCT literature classification model training system based on adversarial training, as shown in Figure 2 , which comprises:

[0135] The data set module 201 is configured to obtain a specified medical literature data set with annotations. The annotations include RCT literature, non-RCT literature, and divergence literature. The divergence literature refers to literature for which multiple annotators disagree on whether the literature is RCT literature.

[0136] The adversarial module 202 is configured to construct an adversarial model based on the literature in the data set that is annotated as divergence literature, and train the adversarial model based on the literature in the data set that is annotated as RCT literature and non-RCT literature.

[0137] The training module 203 is configured to run the adversarial model, generate a sample library, and train a binary classification model based on the sample library. The binary classification model is a machine learning model for identifying whether literature is RCT literature. The proportion of RCT literature and non-RCT literature in the sample library satisfies a preset sample balance condition.

[0138] Embodiments of the present application also provide a terminal device, as shown in the figure. The terminal device 30 includes at least one processor 301, a memory 302, and a computer program 303 stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the above method embodiments are implemented. Figure 3

[0139] The above is only a preferred embodiment of the present application, and does not limit the present application in any way. Any person skilled in the art can make any form of equivalent replacement, modification, or change to the technical solutions and technical content disclosed in the present application without departing from the scope of the technical solutions of the present application, and still fall within the scope of protection of the present application.​

Claims

1. A training method for RCT literature classification model based on adversarial training, characterized in that: The following steps are involved: Obtain a designated medical literature dataset with annotations; wherein the annotations include RCT literature, non-RCT literature, and divergent literature; the divergent literature refers to literature in which multiple annotators have different judgments on whether the literature is an RCT literature; Building an adversarial model based on documents marked as divergent documents in the data set, and training the adversarial model based on documents marked as RCT documents and non-RCT documents in the data set; Running the adversarial model to generate a sample library, and training a binary classification model based on the sample library; wherein the binary classification model is a machine learning model for identifying whether a document is an RCT document, and the ratio of RCT documents to non-RCT documents in the sample library satisfies a preset sample balance condition; the sample balance condition is that the number of RCT documents and non-RCT documents is equal; and all samples in the sample library are labeled; The adversarial model is used to generate new samples that imitate the fuzzy blueprint, or to identify features of the fuzzy blueprint, wherein the fuzzy blueprint refers to the divergent document; The core discrimination rule of the adversarial model is obtained based on the extraction and encoding of fuzzy features, wherein the fuzzy features refer to the features in the divergent documents that cause the annotators to disagree; Extracting the fuzzy features includes: using NLP semantic parsing technology to structurally decompose the divergent documents and identify ambiguous sentence patterns and contradictory text structures; The loss function of the adversarial model includes an annotator disagreement metric, which is used to force the model to focus on the characteristics of the divergent documents during training, so that it can quantify the uncertainty area of ​​the classification boundary; The binary classification model includes a semantic understanding model and a fully connected prediction layer; wherein the semantic understanding model is used to extract features; the fully connected prediction layer is used to predict the probability that a document is an RCT document based on the features extracted by the semantic understanding model, and output a binary classification result based on the probability; The semantic understanding model includes: The first feature extraction layer is used to extract the research framework features of the literature; The semantic understanding model includes: The second feature extraction layer is used to extract the hidden exclusion features of the document; The first feature extraction layer and the second feature extraction layer constitute the first branch of the semantic understanding model; the semantic understanding model also includes a second branch, which is used to output explanatory text of the research framework features and / or the hidden exclusion features; the explanatory text is a natural language text, which is used by researchers to count the logical chains in the research framework features and / or the keywords in the hidden exclusion features during the training process, estimate the effect of the training, and then fine-tune the training process to achieve better model performance.

2. The RCT document classification model training method based on adversarial training according to claim 1 is characterized in that: Outputting a binary classification result according to the probability includes: If it is determined that the probability is greater than the threshold, a binary classification result indicating that the document belongs to an RCT document is output; otherwise, a binary classification result indicating that the document belongs to a non-RCT document is output.

3. The RCT document classification model training method based on adversarial training according to claim 2 is characterized in that: The threshold is a dynamic threshold adjusted according to the year of publication of the literature.

4. A RCT literature classification model training system based on adversarial training, characterized by: include: A dataset module is used to obtain a designated medical literature dataset with annotations; wherein the annotations include RCT literature, non-RCT literature, and divergent literature; the divergent literature refers to literature where multiple annotators have different judgments on whether the literature is an RCT literature; an adversarial module, configured to construct an adversarial model based on documents marked as divergent documents in the dataset, and train the adversarial model based on documents marked as RCT documents and non-RCT documents in the dataset; a training module for running the adversarial model, generating a sample library, and training a binary classification model based on the sample library; wherein the binary classification model is a machine learning model for identifying whether a document is an RCT document, and the ratio of RCT documents to non-RCT documents in the sample library satisfies a preset sample balance condition; the sample balance condition is that the number of RCT documents and non-RCT documents is equal; and all samples in the sample library are labeled; The adversarial model is used to generate new samples that imitate the fuzzy blueprint, or to identify features of the fuzzy blueprint, wherein the fuzzy blueprint refers to the divergent document; The core discrimination rule of the adversarial model is obtained based on the extraction and encoding of fuzzy features, wherein the fuzzy features refer to the features in the divergent documents that cause the annotators to disagree; Extracting the fuzzy features includes: using NLP semantic parsing technology to structurally decompose the divergent documents and identify ambiguous sentence patterns and contradictory text structures; The loss function of the adversarial model includes an annotator disagreement metric, which is used to force the model to focus on the characteristics of the divergent documents during training, so that it can quantify the uncertainty area of ​​the classification boundary; The binary classification model includes a semantic understanding model and a fully connected prediction layer; wherein the semantic understanding model is used to extract features; the fully connected prediction layer is used to predict the probability that a document is an RCT document based on the features extracted by the semantic understanding model, and output a binary classification result based on the probability; The semantic understanding model includes: The first feature extraction layer is used to extract the research framework features of the literature; The semantic understanding model includes: The second feature extraction layer is used to extract the hidden exclusion features of the document; The first feature extraction layer and the second feature extraction layer constitute the first branch of the semantic understanding model; the semantic understanding model also includes a second branch, which is used to output explanatory text of the research framework features and / or the hidden exclusion features; the explanatory text is a natural language text, which is used by researchers to count the logical chains in the research framework features and / or the keywords in the hidden exclusion features during the training process, estimate the effect of the training, and then fine-tune the training process to achieve better model performance.

5. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

7. A computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images