RCT literature labeling method and system based on model consensus

Abstract

The invention provides a model consensus-based RCT literature labeling method and system. The method comprises the following steps: acquiring a literature to be labeled; inputting the to-be-labeled literature into the first model to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM; inputting the to-be-labeled literature into a second model to obtain a second RCT recognition result; wherein the second model is a special machine learning model which is obtained based on sample and label training and is used for RCT literature resolution; if it is determined that the first RCT recognition result is the same as the second RCT recognition result, setting a label of the to-be-labeled literature based on the same first RCT recognition result and second RCT recognition result; wherein the labels comprise RCT literatures and non-RCT literatures. According to the method, labeling with higher classification accuracy (namely, the accuracy is higher, the precision is higher, and the recall rate is better) is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of RCT document annotation, and in particular relates to a RCT document annotation method and system based on model consensus. Background Art

[0002] With the development of medical concepts, the current medical model has shifted from empirical medicine to evidence-based medicine (EBM). EBM, which adheres to the principle that "all clinical decisions should be based on clinical evidence," provides the most robust evidence support and rigorous guidance for clinical research design, and has important guiding significance for both clinical practice and scientific research. The primary evidence carrier of EBM is the systematic review, which has extremely stringent writing requirements. Researchers must systematically search and screen the literature for a specific clinical question to identify the best available clinical evidence, assess the risk of bias, and integrate the results. This process involves systematic searching, literature screening, information extraction, bias risk assessment, and data synthesis. To mitigate the risk of bias inherent in the included literature, systematic review authors typically seek the best available clinical evidence from rigorously designed randomized controlled clinical trials (RCTs). To ensure systematic retrieval, reviewers often use search strategies with extremely high recall (sensitivity) but very low precision (accuracy). This often results in literature screening processes involving thousands of titles and abstracts, the vast majority of which are not RCTs. This consumes considerable time and effort, and results in the writing and updating of systematic reviews being far slower than the pace required for clinical decision-making. One study showed that evaluating a systematic review requires screening an average of 1,781 articles, with an average exclusion rate of 97.1% for irrelevant literature. The average time required to publish a systematic review is 64.3 weeks.

[0003] Although literature screening itself requires strict requirements, is complex and labor-intensive, the intelligent classifiers currently developed for this purpose still have the following drawbacks: 1. Data annotation defects; 2. Annotation noise: The Kappa coefficient of traditional manual annotation is only 0.65 (the ideal value is >0.9); 3. Sample imbalance: Insufficient diversity of non-RCT literature (the existing dataset only covers 42 study types); 4. The recall rate still needs to be further improved and optimized; 5. Although the existing RCT missed literature recall program can further reduce the number of missed RCT literature, the number of non-RCT literature incorrectly marked as RCT is too large (the false positive rate is too high). Summary of the Invention

[0004] The embodiments of the present application provide a method and system for training an RCT document classification model based on adversarial training, which can solve the problem of insufficient accuracy in RCT document classification.

[0005] In a first aspect, the present application provides a method for annotating RCT (Randomized Controlled Clinical Trial) literature based on a model consensus, comprising the following steps: Obtain documents to be annotated; Inputting the document to be annotated into a first model to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM (Large Language Model); Inputting the document to be annotated into a second model to obtain a second RCT identification result; wherein the second model is a dedicated machine learning model for RCT document identification obtained by training based on samples and labels; If it is determined that the first RCT recognition result and the second RCT recognition result are the same, a label for the document to be annotated is set based on the same first RCT recognition result and the second RCT recognition result; wherein the label includes RCT documents and non-RCT documents.

[0006] In a possible implementation of the first aspect, after the step of obtaining the document to be annotated, the method further includes: Standardize the document to be annotated; wherein, the standardization refers to converting non-standard expressions in the document to be annotated into standard semantics based on a preset knowledge graph; the knowledge graph and / or the standard semantics are obtained based on the RCT concept in the Unified Medical Language System (UMLS).

[0007] In a possible implementation of the first aspect, after the step of obtaining the document to be annotated, the method further includes: Perform entity alignment on the document to be annotated and a preset knowledge base; wherein, the entity alignment refers to contextual disambiguation of potential semantic ambiguities.

[0008] In a possible implementation of the first aspect, the step of inputting the document to be annotated into the first model includes: Inputting the document to be annotated into the first model based on a preset prompt; The preset prompt is obtained based on dynamic iteration of rules; the dynamic iteration based on rules includes: Get prompt rules; The prompt rules are grouped to obtain a passing rule set, a prompt rule set, and a warning rule set; wherein when elements in the passing rule set are satisfied, the document to be annotated is identified as an RCT document; when elements in the prompt rule set are satisfied, the confidence that the document to be annotated is identified as an RCT document is increased; when elements in the warning rule set are satisfied, the confidence that the document to be annotated is identified as an RCT document is decreased; A prompt is constructed according to the pass rule set, the prompt rule set, and the warning rule set; wherein the prompt satisfies: the rule determination priorities of the pass rule set, the warning rule set, and the prompt rule set are in descending order; Running the first model based on the prompt and a preset document sample set to obtain a pre-annotated result with a thought chain; updating the prompt rule according to at least one of the thought chain corresponding to the bad case in the pre-annotated result, the rule count of the prompt rule, and the false positive rate of the prompt rule; wherein the updating includes at least one of adjusting the weight of the prompt rule, adjusting the rule of the prompt rule, replacing the rule of the prompt rule, and adjusting the set to which the prompt rule belongs; Based on the updated prompt rule, return to the step of constructing the prompt, and perform an iterative operation until a preset iteration stop condition is met.

[0009] In a possible implementation of the first aspect, the step of inputting the document to be annotated into the second model includes: If it is determined that the first RCT identification result is a non-RCT document, the document to be annotated is converted into a structured mapping field table in a specified format, and the structured mapping field table in the specified format is input into the second model.

[0010] In a possible implementation of the first aspect, the step of obtaining the document to be annotated includes: If it is determined that the preset crawling conditions are met, then unannotated documents are crawled from the specified database as the documents to be annotated; The marking priority of the document to be marked is set according to a preset task scheduling condition; wherein the task scheduling condition includes: the marking priority of the document to be marked is positively correlated with the impact factor of the document to be marked.

[0011] The advantages of the embodiments of the present application compared with the prior art are: The consensus of the first and second models is used to complete the RCT annotation of the literature, combining the breadth coverage of the first model and the domain accuracy of the second model, thereby achieving annotation with higher classification accuracy (i.e., better accuracy, precision, and recall).

[0012] In a second aspect, the present application provides an RCT document annotation system based on model consensus, including: The acquisition module is used to obtain the documents to be annotated; A first model module is configured to input the document to be annotated into a first model to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM; A second model module is configured to input the document to be annotated into a second model to obtain a second RCT identification result; wherein the second model is a dedicated machine learning model for RCT document identification obtained by training based on samples and labels; The consensus module is used to determine that the first RCT recognition result and the second RCT recognition result are the same, and then set a label for the document to be annotated based on the same first RCT recognition result and the second RCT recognition result; wherein the label includes RCT documents and non-RCT documents.

[0013] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the model consensus-based RCT document annotation method described in any one of the first aspects above is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the RCT document annotation method based on model consensus described in any one of the first aspects above.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the RCT document annotation method based on model consensus as described in any one of the first aspects above.

[0016] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the workflow diagram of the RCT literature annotation system based on model consensus.

[0018] Figure 2 A diagram of the system operation architecture provided in the embodiment of the present application; Figure 3 The comparison chart of some literature discrimination results before and after the standardization operation; Figure 4 This is a framework diagram of the RCT literature annotation system based on model consensus; Figure 5 This is a framework diagram of the terminal device. DETAILED DESCRIPTION

[0019] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0020] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0022] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0024] The model consensus-based RCT document annotation method and system of the present invention will be described in more detail below with reference to a schematic diagram, which shows a preferred embodiment of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the beneficial effects of the present invention.

[0025] The present application embodiment provides a RCT literature annotation method based on model consensus, comprising the following steps: Step 102: Obtain documents to be annotated; Step 104: input the document to be annotated into a first model to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM; Step 106: Input the document to be annotated into a second model to obtain a second RCT identification result; wherein the second model is a dedicated machine learning model for RCT document identification obtained by training based on samples and labels; Step 108: If it is determined that the first RCT recognition result and the second RCT recognition result are the same, a label for the document to be annotated is set based on the same first RCT recognition result and the second RCT recognition result; wherein the label includes RCT documents and non-RCT documents.

[0026] In this embodiment, the execution order of step 104 and step 106 is not limited by the step number or the recording order. Step 104 and step 106 can be executed simultaneously, or step 104 can be executed first and then step 106, or step 106 can be executed first and then step 104.

[0027] One of the key points of this embodiment is the dual-model consensus architecture implemented by the cooperation of step 104 and step 106. The architecture based on model consensus can significantly improve the accuracy of annotation.

[0028] At the same time, as indicated by the serial numbers of step 104 and step 106, there may be other steps between the two, such as introducing more models to optimize the accuracy of the consensus through step 105 (the same is true for other steps, and other steps may be introduced before, after or between the steps of this embodiment).

[0029] Step 102 is the starting point of the process of this embodiment. In an optional embodiment, the documents to be annotated are derived from multiple source databases such as PubMed, EMBase, Scopus, Web of Science, and the Cochrane Library. These documents are obtained through Python scripting tools or institutional subscription agreements.

[0030] Optionally, these can be deduplicated and formatted normally, for example, by converting PDF, XML, and other formats into JSON-LD, a structured data format that supports semantic annotation and cross-platform compatibility. The field structure of the JSON-LD compartment can be defined, including the title, abstract, keywords, and the methods and results sections of the full paper.

[0031] Data quality control includes integrity verification (such as checking key fields such as PMID and DOI) and manual sampling to ensure that data quality meets the requirements of subsequent processing.

[0032] In particular, this embodiment also provides an implementation of parallel processing, specifically including: The form is split according to the size of the form imported into the system (the system design in this embodiment supports forms with more than 65,535 lines), usually into groups of 6,000 lines. Multiple forms are constructed within the system, and two models are called in parallel for annotation. This method increases the maximum capacity of the RCT literature screening project while reducing the resource load required for system operation. After annotation, the form is then integrated according to the built-in preset index column and the entire form is output; That is, after step 102 and before step 104 and / or step 106, the method further includes: An information table of the documents to be annotated is constructed, and the information table is divided based on the number of the documents to be annotated to obtain multiple tables; wherein the multiple tables are used to run the first model and the second model in parallel.

[0033] The following is an implementation plan actually performed by the applicant.

[0034] It is worth noting that the processes corresponding to the "implementation methods actually implemented by the applicant" in this and subsequent examples are optimized, preferred processes based on the solutions in the various examples. Their purpose is to better illustrate the refinement and expansion of the examples and does not constitute a limitation of the examples. Furthermore, the processes corresponding to the "implementation methods actually implemented by the applicant" provide specific performance comparison data (see subsequent description), providing a better data foundation for demonstrating the advantages of each example.

[0035] The process corresponding to "the implementation method actually implemented by the applicant" is implemented based on the following overall ideas.

[0036] A web platform for automated screening of RCT literature was built. Based on the datasets from different sources analyzed and collated by our team in the early stage, customized small models were trained using technologies such as the Large Language Model (LLM) and BERT. By deploying the Deepseek series of LLMs (R1 / V3 versions) to build a primary screening layer, and using its general semantic understanding capabilities to achieve broad coverage of document titles / abstracts (identifying implicit grouping information such as "blinded allocation"), combined with dynamically optimized structured prompts (promptB version), the false negative rate is controlled below 5%; for suspicious documents screened by LLM, a customized small model is used for secondary judgment, and a domain-specific high-sensitivity model (with dynamically adjustable sensitivity parameters) is used to capture RCT features that may be missed by LLM. The goal is to achieve an RCT missed screening rate of less than 0.01% (false negative rate ≤ 0.01%), which is more than 99.5% lower than the traditional small model (typical missed screening rate of 2-5%), and the rate of mislabeled RCT documents is less than 10% (false negative rate ≤ 10%), thereby serving clinical evidence-based research.

[0037] Among them, concepts such as large language model (such as Deepseek), LLM, coarse granularity, and pre-screening correspond to the first model of this embodiment, and concepts such as BERT, Robotsearch, small model, fine granularity, and correction correspond to the second model of this embodiment.

[0038] like Figures 1 and 2 As shown, a number of experts screened out 780 eligible included documents and 7545 excluded documents according to the inclusion and exclusion criteria, totaling 8325 documents. They were imported into the local client system and the annotation button was clicked. The data was first cleaned and organized, and then the API of Deepseek-V3 large model of Deepseek was called for the first analysis and annotation. A table with 1330 RCT documents (including 764 included and 566 excluded documents determined by experts a priori) and 6995 non-RCT documents (including 16 included and 6979 excluded documents determined by experts a priori) was identified and stored in memory. After the initial annotation is completed, the server-side Robotsearch API is called for secondary annotation based on the result of the newly added Deepseek column labeled as NonRCT. A table with 1,446 RCT documents (including 10 included and 1,436 excluded documents based on expert prior judgment) and 5,549 non-RCT documents (including 6 included and 5,543 excluded documents based on expert prior judgment) marked out is identified and stored in the built-in storage. This forms the final result table with RCT and NonRCT marked out, and provides an export interface.

[0039] The applicant is committed to building an intelligent and automated screening system for randomized controlled trial (RCT) literature based on a large language model (LLM). The system focuses on three main technical lines: hybrid architecture design, dynamic rule engine, and automated annotation system. The system aims to address the problems of traditional RCT identification methods, such as high false positives and false negatives, low efficiency in resolving rule conflicts, and high manual annotation costs. Specifically: 1. LLM-driven RCT hybrid screening architecture optimization and validation (1) Reconstruction of the hierarchical model coordination mechanism To address the difficulty of balancing speed and accuracy in traditional single-layer models, a three-layer architecture of "coarse-grained LLM screening - fine-grained small model correction - dynamic feedback learning" is proposed: First layer: LLM semantic pre-screening network Using a multi-task joint training strategy, RCT identification is broken down into extracting key title elements (such as "randomization" and "double-blind"), parsing the logical structure of the abstract (such as segmenting the study objectives, methods, and results), and constructing a group relationship map (clarifying the semantic correspondence between the intervention and control groups). By designing heterogeneous prompt templates (such as "Determine whether it is an RCT: Input title + abstract, output decision rationale in JSON format"), the LLM is forced to output a structured reasoning process, improving interpretability. Preliminary testing shows that this architecture reduces the false negative rate on the PubMed dataset from 9% to 5%.

[0040] Second layer: domain adaptation of medical correction models A lightweight domain adaptation model (DeBERTa-v3) was developed based on the BERT architecture, with the number of parameters controlled within 500M. A two-stage training scheme was used: Pre-training: Masked language modeling (MLM) was performed based on 1,330 retracted RCT papers from our team's previous retraction literature research. Domain-specific word vectors were learned (e.g., "random assignment" outperforms its synonym "random grouping"). Retracted non-RCT papers from the Retraction Watch Database were also introduced to construct adversarial examples for training. Fine-tuning phase: Using a labeled pediatric adverse drug reaction research literature dataset (with a sample size of 13,564 RCTs and 41,664 non-RCTs) combined with comparative learning, the model's ability to distinguish negative samples (such as case-control studies and retrospective cohort studies) was enhanced.

[0041] Layer 3: Active Learning Loop for Conflicting Samples We established an uncertainty-based sample selection mechanism to dynamically mark documents with significant discrepancies between LLM and MED-BERT predictions (e.g., confidence difference > 0.3) and push them to the expert annotation queue. With each round of iterative updates, we aim to improve the model's recognition accuracy for borderline samples by 15%-20%.

[0042] (2) Semantic mapping technology for unstructured documents Develop a knowledge graph-based semantic standardization module for non-standard expressions in medical literature (e.g., “two-arm parallel design” versus “two-group randomized controlled trial”): Extract RCT-related concepts (such as NCT codes and CONSORT terms) from UMLS (Unified Medical Language System) to build an RCT ontology knowledge base, covering 27 key attributes and 142 semantic relationships; During the model inference phase, entities are aligned between the original text and the knowledge base, enabling contextual disambiguation of potential semantic ambiguities (e.g., "random" might refer to "sampling method" rather than "grouping method"). The team's preliminary experiments have shown that this technology improves the correct recognition rate of complex documents by 22%.

[0043] That is to say, based on step 102, in a preferred embodiment: After the step of obtaining the document to be annotated, the method further includes: Standardize the document to be annotated; wherein, the standardization refers to converting non-standard expressions in the document to be annotated into standard semantics based on a preset knowledge graph; the knowledge graph and / or the standard semantics are obtained based on the RCT concept in the Unified Medical Language System (UMLS).

[0044] In another preferred embodiment: After the step of obtaining the document to be annotated, the method further includes: Perform entity alignment on the document to be annotated and a preset knowledge base; wherein, the entity alignment refers to contextual disambiguation of potential semantic ambiguities.

[0045] 2. Dynamic rule engine conflict resolution and enhanced explainability (1) Hierarchical modeling of medical rule graphs Based on the CONSORT (Consolidated Standards of Reporting Trials) statement and PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) standards, a four-dimensional rule evaluation system was constructed: Table 1 Four-dimensional rule evaluation system Rule Level Decision logic Example Priority L1 Title directly through "Randomized double-blind controlled trial" 5 L2 Summary of key evidence chains Includes randomization, group comparison, and outcome indicators 4 L3 Implicit prompt word mining "Blind assessment" and "allocation concealment" 3 L4 Journal authority compensation The New England Journal of Medicine and other RCT marking compensation 2 A weighted decision-making mechanism (weights determined by expert scoring) enables dynamic adjustments when rules conflict. For example, when an L1 rule is triggered but an L2 warning signal is present, the system automatically assigns the document to the manual review queue, ensuring 100% review of high-risk documents.

[0046] (2) Iterative Evolution Path of Dynamic Prompt Adopting a closed-loop strategy of "rule-based initial design - data-driven optimization - manual verification": Construction of the initial rule zero-shot prompt: Integrating sources such as the Cochrane Handbook and the JAMA (Journal of the American Medical Association) submission guidelines, 32 core rules were extracted (e.g., "If the abstract contains 'double-blind design' and does not mention 'retrospective,' it should be marked as an RCT"), and constructing the initial promptA based on expert knowledge. Three core elements were clearly defined: (1) the role was defined as a professional research assistant, (2) the goal was to focus on the identification of RCT characteristics of population groups in clinical trials, and (3) the output was mandatory in JSON format. Its judgment logic established a three-level priority system: direct evidence (e.g., titles containing "safety and efficacy") > warning signs (e.g., "retrospective") > prompt words (e.g., "double blind"), and it was required that fuzzy grouping descriptions (not explicitly "random" but mentioning grouping) should still be marked as possible RCTs.

[0047] An iterative cycle was initiated using 200 expert-annotated data points. PromptA first generated pre-annotated results with thought chains, added a few shot content, and further constructed and designed the following key rules based on the CoT of the bad case: <direct pass: very very very likely a RCT paper, you see this you goahead label it a RCT> - "safety and efficacy" in title - the term "comparison" or same meaning saying in title or abstract - the term "control" or "control group" or same meaning saying intitle or abstract - the term "group" or same meaning saying in title or abstract - used causal inference to make the conclusion, because journals most of the time only publish it when accompanied by RCT <hint: RCT paper often use such words:> - assign, doubly, double blind, allocated, trial, efficiency, efficacy <red flags: Non-RCT often use such words> - "we searched", "questionarre", "survey", "database", "test tube / invitro experiment", "cross-sectional", "retrospective", "animal", "casecontrol" This final iteration generates a promptB version with an accuracy ≥ 97%.

[0048] Dynamic rule update mechanism: Monthly statistics are collected on the number of rule triggers and false positive rates. Rules with false positive rates greater than 10% are downgraded or replaced. For example, after discovering that the word "comparison" frequently appears in cross-sectional studies, it is removed from the L2 rule.

[0049] That is, based on step 104, in a preferred embodiment: The step of inputting the document to be annotated into the first model comprises: Inputting the document to be annotated into the first model based on a preset prompt; The preset prompt is obtained based on dynamic iteration of rules; the dynamic iteration based on rules includes: Get prompt rules; The prompt rules are grouped to obtain a passing rule set, a prompt rule set, and a warning rule set; wherein when elements in the passing rule set are satisfied, the document to be annotated is identified as an RCT document; when elements in the prompt rule set are satisfied, the confidence that the document to be annotated is identified as an RCT document is increased; when elements in the warning rule set are satisfied, the confidence that the document to be annotated is identified as an RCT document is decreased; A prompt is constructed according to the pass rule set, the prompt rule set, and the warning rule set; wherein the prompt satisfies: the rule determination priorities of the pass rule set, the warning rule set, and the prompt rule set are in descending order; Running the first model based on the prompt and a preset document sample set to obtain a pre-annotated result with a thought chain; updating the prompt rule according to at least one of the thought chain corresponding to the bad case in the pre-annotated result, the rule count of the prompt rule, and the false positive rate of the prompt rule; wherein the updating includes at least one of adjusting the weight of the prompt rule, adjusting the rule of the prompt rule, replacing the rule of the prompt rule, and adjusting the set to which the prompt rule belongs; Based on the updated prompt rule, return to the step of constructing the prompt, and perform an iterative operation until a preset iteration stop condition is met.

[0050] Among them, the dynamic rule engine can be understood as an engine for executing dynamic iteration based on rules; the rule map, evaluation system, hierarchical modeling, and rule hierarchy can be understood as the grouping and collection division of prompt rules; promptA can be understood as the prompt before or during the iteration, and promptB can be understood as the prompt after the iteration is completed.

[0051] In particular, rule-based adjustment can also consider the parallel operation analysis of conflicting rules and mutually exclusive rules.

[0052] 3. Engineering Implementation and Effectiveness Verification of the Automated Labeling System (1) Coupling integration of large language model modules and specialized small models A three-level processing flow is adopted: first, the official Deepseek-R1 API is called (with parameters set to temperature=0 and top_p=1) for initial literature screening. Then, a special small model is used to conduct secondary judgment on the returned Non-RCT results. The small model is encapsulated as a Docker image and provides a RESTful interface service. The system design includes an API exception retry mechanism (maximum 10 loop judgments), structured prompt version management (promptA / promptB), and strict result format verification (mandatory JSON output). By combining the breadth coverage of the large model with the domain precision of the small model, a hierarchical filtering of RCT documents with an accuracy rate of 97% is achieved.

[0053] That is, based on step 106, in a preferred embodiment: The step of inputting the document to be annotated into the second model comprises: If it is determined that the first RCT identification result is a non-RCT document, the document to be annotated is converted into a structured mapping field table in a specified format, and the structured mapping field table in the specified format is input into the second model.

[0054] (2) Construction of RAG knowledge base for annotation system Data pipeline: Automatically crawl new documents from databases such as PubMed every day, remove duplicates, and standardize the format (JATSXML to JSON) before entering the annotation pool; Task scheduling: Dynamically allocate annotation tasks, giving priority to high-impact journal articles or model dispute samples; Construction of RAG knowledge base: The dynamically annotated literature data is vectorized and included in the database. When the user submits data for automatic screening next time, the system will first call the server RAG knowledge base to perform matching operations and annotate the user data based on the annotation results of the vector database.

[0055] That is to say, based on step 108, in a preferred embodiment: The step of obtaining the document to be annotated includes: If it is determined that the preset crawling conditions are met, then unannotated documents are crawled from the specified database as the documents to be annotated; The marking priority of the document to be marked is set according to a preset task scheduling condition; wherein the task scheduling condition includes: the marking priority of the document to be marked is positively correlated with the impact factor of the document to be marked.

[0056] The following will describe the applicant's invention process based on the process corresponding to the "implementation method actually operated by the applicant" to illustrate the advantages of each embodiment of the present application through the scheme iteration and data situation.

[0057] The applicant adopts a four-stage promotion plan of "basic theory and technology research and development, core system construction, multi-center verification, and product implementation", with a total cycle of 24 months, integrating three major technical systems: natural language processing, clinical research methodology, and software engineering. The technical implementation stages are as follows: Phase 1 (0-6 months): Establish theoretical and data foundations, complete the construction of RCT identification rule system and development of annotation system; Phase 2 (July-December): System prototype development to achieve dynamic tuning of the hybrid architecture and engineering deployment of the rule engine; Phase 3 (January 13-20): Multi-center clinical efficacy verification, conducting real-world testing in collaboration with five tertiary hospitals and the Cochrane Collaboration Group; Phase 4 (21-24 months): Product iteration and regulatory approval, and completion of medical device software (SaMD) certification.

[0058] The final plan focuses on: 1. Heterogeneous data format dynamic conversion module • Standardized input interface Develop a literature data preprocessor to convert and merge multiple source literature forms (.csv, .xls, .xlsx, .bib) into the RobotSearch-specific RIS format: Structured field mapping: Title and summary are forced to align to fixed fields (such as<randomization_block> tags) and remove blank lines between paragraphs.

[0059] 2. Server-client collaborative automatic annotation system •Server functional architecture RobotSearch microservice deployment: Encapsulated as a Docker container, providing a RESTful API interface (response time < 200ms) Support batch document processing (up to 100 documents per request, throughput ≥ 50 documents / second) Annotation state synchronization engine: Maintain the global annotation queue in real time to avoid repeated submissions from clients (conflict detection accuracy > 99%) •Client interaction design One-time annotation trigger logic: The client automatically calls the RobotSearch service for the LLM initial screening results (such as documents with the nonRCT_2 label) Multi-format document batch processing verification: Input data: Test set: 1000 mixed-format literature forms (title + abstract) (400 in .csv format, 300 in .xlsx format, and 300 in .ris format).

[0060] Processing flow: Data format conversion: uniformly mapped to RobotSearch input specifications through the preprocessing module, and fixed 23 format errors.

[0061] Server call: The client sends a JSON request to the server RobotSearch microservice (including the document ID and the cleaned title + abstract).

[0062] Table 2 Performance indicators

[0063] Table 3 Comparison data of key effects Comparison Dimension Traditional server architecture This technical solution Improvement Format processing efficiency Manual conversion takes 2.5 minutes per article Automatic conversion takes 0.3 seconds per article 500 times Resource usage 8GB of memory required for local deployment Server load <1% CPU / request 97% reduction Labeling accuracy Single model annotation F1 0.81 Collaborative annotation F1 0.98 +21% The implementation process specifically includes: 1.1 Basic data construction and standard formulation (1) Multi-source data collection solution Table 4 Data source configuration source Data volume type How to obtain PubMed 5 million articles Structured metadata + abstract Python script tool acquisition EMbase 5 million articles Structured metadata Python script tool acquisition Scopus 5 million articles Structured metadata Python script tool acquisition Web of science 5 million articles Structured metadata Python script tool acquisition Cochrane Library 100,000 systematic reviews Full text PDF Institutional Subscription Agreement Heterogeneous data processing: Develop a unified conversion tool to standardize PDF, XML, HTML and other formats into JSON-LD format, including the following fields: <text> { "title": {text, lang: "zh / en"}, "abstract": {structured: [PICO feature extraction results]}, "keywords": {original: [], normalized: [UMLS mapping term]}, "fulltext_section": {methods: [], results: []} } Data quality control: Completeness check: Eliminate or manually complete documents that are missing key fields (such as PMID, DOI); An expert and experienced member of the research team randomly selected 100 documents and evaluated the quality of the information obtained.

[0064] (2) Gold Standard Data Annotation Specification Marking rules formulation: Two research assistants (with ≥3 years of experience in data annotation) in evidence-based medicine literature screening developed the "RCT Literature Annotation Operation Manual," which defines three categories: Include category (directly through evidence): The title contains "randomized controlled" and the abstract clearly states the grouping method Maybe (full text review required): The abstract mentions "random allocation" but does not describe the implementation of the blinding method or the description is ambiguous (e.g., "random sampling method was used") Exclude class (automatic exclusion): explicitly declare "retrospective study" or "single-group design" Annotator training system: Theoretical training: Online learning of CONSORT standards and annotation specifications (18 hours); Practical assessment: Complete 100 sample annotations with a Kappa coefficient ≥ 0.8; Continuous monitoring: 10% of the labeled results will be randomly inspected every month, and those with an accuracy rate < 95% will have their permissions suspended.

[0065] 1.2 Core Technology R&D Plan (1) Implementation path of hybrid architecture engineering Table 5 Component development sequence Module Technical Solution Development cycle Technical verification indicators Large Language Model Deepseek API encapsulation + custom prompt template iterative optimization 2 months Title + abstract recognition accuracy ≥ 98% Special small model Fine-tuning DeBERTa-v3 based on self-training data 4 months False positive rate ≤ 3% Rules Engine Drools rule base + dynamic priority algorithm 4 months Rule coverage ≥ 90% Model Portfolio Based on LLM and self-training models, the interface combination is integrated to form the core engine of screening and recall automation 2 months The false positive rate of the combined model is ≤0.01% Front-end interface Vue3+ECharts Visualize Decision Path 2 months Response time ≤ 500ms Key algorithm implementation details: Use Bayesian optimization method to adjust the classification threshold.

[0066] (2) Dynamic rule engine development Project implementation steps: Designing rule syntax: Developing a dynamic rule engine first requires defining clear rule classification and priority. Rules should be divided into direct passing criteria (e.g., those with titles containing "safety and efficacy"), prompting rules (e.g., "random assignment" and "control group"), and warning rules (e.g., "cross-sectional study" and "animal experiment"). The execution priority should also be set (direct criteria > warning signals > prompting rules). Then, a structured grammar was designed, using an extensible markup language (e.g., XML-style rule groups) to support logical operations, and dynamic optimization mechanisms were introduced, such as looping to determine uncertain results and expert review feedback to modify rules. At the same time, it needs to work in collaboration with the AI model. The front-end rule engine processes explicit features, combines LLM semantic analysis capabilities, and supports dynamic adjustment of sensitivity and specificity. Finally, strict verification is carried out to ensure that the rule accuracy rate is ≥95%, and hot update and rollback of rules are achieved through version management; Performance optimization solution: ① Rule pre-compilation: compile high-frequency rules (trigger rate > 1%) into C++ extension modules (increase speed by 20 times); ② Parallel execution: Allocate independent thread pools to mutually exclusive rule groups (maximum concurrency = number of CPU cores × 2).

[0067] (3) Construction of automated annotation system Table 6 Technology stack selection Components Technical Solution Selection basis Data storage MongoDB Sharded Cluster Support efficient retrieval of unstructured data Task Scheduling Airflow DAG Visual pipeline monitoring Model training PyTorch Lightning Support multi-GPU distributed training Annotation process optimization: Develop a "two-step quality control method" to ensure data reliability: Machine automated screening: Use pre-trained models to filter out obvious mislabeling (e.g. mislabeling "random sampling" as RCT); Expert testing: Domain experts proofread and confirm 0.5% of high-weight samples (information entropy > 2.0).

[0068] 1.3 Effectiveness Verification and Results Transformation Plan (1) Multi-level verification framework design Table 7 Technical indicator verification Test Type Dataset Evaluation Metrics Expected goals Unit Testing Adversarial Examples (n=11,142) Sensitivity / specificity >95% Integration Testing PubMed full library subset (n=80,000) Processing speed >1000 articles / second Clinical validation Multicenter real-world literature (n=15,832) Reduction in manual review ≥70% Clinical validation implementation steps: Double-blind controlled trials were conducted in collaboration with Shanghai Pulmonary Hospital, Renji Hospital, and West China Hospital: Experimental groups: Experimental group: pre-screened using this system + manual review; Control group: traditional fully manual screening (Covidence system); Evaluation indicators: Main indicators: missed detection rate (comparison with standard answers); Secondary indicator: total time spent on literature screening (hours).

[0069] System reliability verification steps: The screening efficacy of the system was evaluated using evidence-based medicine methods, including: ① Conduct a retrospective analysis of the missed literature, track the meta-analysis and systematic review (SR) literature cited by them, and compare the consistency of the main indicators (such as effect size, confidence interval) and conclusions of the SR before and after excluding the missed literature; ② If the key indicators do not change significantly (e.g., p-value shift < 0.05 or conclusion reversal), the reliability of the system in the evidence-based chain is verified; ③ Continue to combine expert annotation (100% accuracy as the gold standard) with the chain of thought (CoT) generated by LLM to analyze the causes of missed detections, optimize the rule base (such as expanding implicit RCT feature words such as "group" and "random") and the prompt word scheme (iterate prompt until the accuracy rate reaches the target) to ensure the balance between sensitivity and specificity of the system.

[0070] (2) Risk management plan Table 8 Technical risk response Risk Type Coping strategies Insufficient data Manually obtain training sets + purchase labeled data sets Model bias Introducing / expanding adversarial training datasets (Exclude) and using domain adaptation techniques (DANN) Rule lag Establish a monthly CONSORT standards tracking mechanism The advantages of the embodiments of the present application are: • Data format fragmentation problem • Traditional methods require manual adjustment of input formats, while this technology achieves seamless integration of multi-source data through dynamic conversion modules; •Hybrid annotation efficiency bottleneck • Manually transferring annotation results takes 30% longer, while this technology reduces manual intervention by 70% through automated flow mechanisms. • The original annotation solution became sluggish after the automated annotation part reached 40,000 forms, and work efficiency was difficult to optimize and improve. The new solution achieved collaborative computing by calling the server API through the local client, breaking through the bottleneck of 40,000 forms processing. The key effect comparison data are shown in Table 9.

[0071] Table 9 Comparative data of key effects Comparison Dimension Traditional server architecture This technical solution Improvement Format processing efficiency Manual conversion takes 2.5 minutes per article Automatic conversion takes 0.3 seconds per article 500 times Resource usage 8GB of memory required for local deployment Server load <1% CPU / request 97% reduction Labeling accuracy Single model annotation F1 0.81 Collaborative annotation F1 0.98 +21% In particular, for standardization operations, the traditional view is that this operation can speed up model training, compatible algorithms, and slightly improve model performance.

[0072] Different from the aforementioned traditional concepts, in the present application scheme, the effects brought about by standardized operations are completely different.

[0073] The following provides performance comparison data before and after standardization. By cleaning the documents input to the second model, removing blank lines, optimizing document formatting, and converting to standard semantics, the recall rate of the model can be significantly improved. The patent applicant's research found that the second model was used to judge 549 RCT documents and 3599 non-RCT documents, and the results are as follows: Confusion matrix of the performance of the second model before literature normalization: Predicted to be RCT Predicted to be non-RCT Total (actual) Actually RCT TPrct = 164 FNrct = 385 549 Actually non-RCT FPrct = 548 TNrct = 3051 3599 Total (forecast) 712 3436 4148 Performance confusion matrix of the second model after literature normalization:

[0074] The relevant numerical analysis form is as follows: index Before document standardization After standardization of the literature Accuracy (ACC) 0.7751 0.6878 Precision 0.2303 0.2971 Recall 0.2987 0.9945 F1 score (F1rct) 0.2601 0.4575 Specificity (Specificity non−rct) 0.8477 0.641 False positive rate (FPRnon−rct) 0.1523 0.359 False Negative Rate (FNRrct) 0.7013 0.0055 This patent focuses on the screening of randomized controlled trial (RCT) literature in the medical field. The goal is to eliminate non-RCT literature while retaining as much RCT literature as possible. The above data results show that standardization of literature can greatly improve the recall rate and F1 value of the literature, and this increase in recall rate and F1 value can better achieve the aforementioned goal of RCT line annotation.

[0075] As can be seen, considering only model performance metrics, ACC even decreased after standardization, significantly differing from conventional wisdom—that is, some model performance metrics decreased while others improved. Although standardization resulted in a decrease in some performance metrics, the impact of these decreases on the purpose of this application—the annotation of RCT documents—was less than that of the increases. Furthermore, the magnitude of the improvements was significant, indicating that the benefits of standardization exceeded expectations, achieving technical effects unexpected to those skilled in the art.

[0076] This unexpected technical effect is mainly reflected in the recall rate, which increased from 0.2987 before the standardization operation to 0.9945 after the standardization operation. After the standardization operation, the room for improvement in the recall rate was almost compressed to the extreme.

[0077] At the same time, this improvement in recall rate did not significantly sacrifice the overall performance of the model, as can be seen from the change in F1 value (0.2601 to 0.4575). Although the ACC has decreased, from the perspective of the purpose of RCT document annotation, the standardized model prioritizes the retention of almost all RCT documents. On this basis, the task of eliminating non-RCT documents is further considered.

[0078] In other words, from the perspective of the performance of the binary classification model alone, the benefits of standardization are still open to discussion. However, for the application scenario of this application, the benefits of standardization (almost all RCT literature can be retained) are exactly what we urgently need (corresponding to the first priority), while its shortcomings (the task of eliminating non-RCT literature) are reflected in the second priority task of this scenario.

[0079] Figure 3 The UI of the embodiment of the present application deployed locally by the applicant is shown, wherein the comparison of some document identification results before and after the standardization operation is shown. Figure 3 The improvement in recall can be seen more intuitively, that is, the literature that was misidentified before standardization can be correctly identified, and almost all the literature that was misidentified as non-RCTs (when no standardization operation was adopted) but was actually RCTs can be corrected through standardization.

[0080] The present application also provides a RCT document annotation system based on model consensus, such as Figure 4 Shown, including: An acquisition module 201 is used to acquire documents to be annotated; The first model module 202 is configured to input the document to be annotated into a first model 204 to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM; The second model module 203 is configured to input the document to be annotated into the second model 205 to obtain a second RCT recognition result; wherein the second model is a dedicated machine learning model for RCT document identification obtained by training based on samples and labels; The consensus module 206 is configured to determine that the first RCT recognition result and the second RCT recognition result are the same, and then set a label for the document to be annotated based on the same first RCT recognition result and the second RCT recognition result; wherein the label includes RCT documents and non-RCT documents.

[0081] The present application also provides a terminal device, such as Figure 5 As shown, the terminal device 30 includes: at least one processor 301, a memory 302, and a computer program 303 stored in the memory and executable on the at least one processor, and the processor implements the steps of any of the above method embodiments when executing the computer program.

[0082] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.< / text>

Claims

1. A RCT literature annotation method based on model consensus, characterized by: The following steps are involved: Obtain documents to be annotated; Inputting the document to be annotated into a first model to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM; Inputting the document to be annotated into a second model to obtain a second RCT identification result; wherein the second model is a dedicated machine learning model for RCT document identification obtained by training based on samples and labels; If it is determined that the first RCT recognition result and the second RCT recognition result are the same, a label for the document to be annotated is set based on the same first RCT recognition result and the second RCT recognition result; wherein the label includes RCT documents and non-RCT documents.

2. The RCT document annotation method based on model consensus according to claim 1, characterized in that: After the step of obtaining the document to be annotated, the method further includes: Standardizing the document to be annotated; wherein, standardization refers to converting non-standard expressions in the document to be annotated into standard semantics based on a preset knowledge graph; The knowledge graph and / or the standard semantics are obtained based on the RCT concept in the Unified Medical Language System (UMLS).

3. The RCT document annotation method based on model consensus according to claim 1 or 2, characterized in that: After the step of obtaining the document to be annotated, the method further includes: Perform entity alignment on the document to be annotated and a preset knowledge base; wherein, the entity alignment refers to contextual disambiguation of potential semantic ambiguities.

4. The RCT document annotation method based on model consensus according to claim 1, characterized in that: The step of inputting the document to be annotated into the first model comprises: Inputting the document to be annotated into the first model based on a preset prompt; The preset prompt is obtained based on dynamic iteration of rules; the dynamic iteration based on rules includes: Get prompt rules; The prompt rules are grouped to obtain a passing rule set, a prompt rule set, and a warning rule set; wherein when elements in the passing rule set are satisfied, the document to be annotated is identified as an RCT document; when elements in the prompt rule set are satisfied, the confidence that the document to be annotated is identified as an RCT document is increased; when elements in the warning rule set are satisfied, the confidence that the document to be annotated is identified as an RCT document is decreased; A prompt is constructed according to the pass rule set, the prompt rule set, and the warning rule set; wherein the prompt satisfies: the rule determination priorities of the pass rule set, the warning rule set, and the prompt rule set are in descending order; Running the first model based on the prompt and a preset document sample set to obtain a pre-annotated result with a thought chain; updating the prompt rule according to at least one of the thought chain corresponding to the bad case in the pre-annotated result, the rule count of the prompt rule, and the false positive rate of the prompt rule; wherein the updating includes at least one of adjusting the weight of the prompt rule, adjusting the rule of the prompt rule, replacing the rule of the prompt rule, and adjusting the set to which the prompt rule belongs; Based on the updated prompt rule, return to the step of constructing the prompt, and perform an iterative operation until a preset iteration stop condition is met.

5. The RCT document annotation method based on model consensus according to claim 1, characterized in that: The step of inputting the document to be annotated into the second model comprises: If it is determined that the first RCT identification result is a non-RCT document, the document to be annotated is converted into a structured mapping field table in a specified format, and the structured mapping field table in the specified format is input into the second model.

6. The RCT document annotation method based on model consensus according to any one of claims 1, 2, 4, and 5, characterized in that: The step of obtaining the document to be annotated includes: If it is determined that the preset crawling conditions are met, then unannotated documents are crawled from the specified database as the documents to be annotated; The marking priority of the document to be marked is set according to a preset task scheduling condition; wherein the task scheduling condition includes: the marking priority of the document to be marked is positively correlated with the impact factor of the document to be marked.

7. A RCT literature annotation system based on model consensus, characterized by: include: The acquisition module is used to obtain the documents to be annotated; A first model module is configured to input the document to be annotated into a first model to obtain a first RCT recognition result; wherein the first model is a general large language model based on LLM; A second model module is configured to input the document to be annotated into a second model to obtain a second RCT identification result; wherein the second model is a dedicated machine learning model for RCT document identification obtained by training based on samples and labels; The consensus module is used to determine that the first RCT recognition result and the second RCT recognition result are the same, and then set a label for the document to be annotated based on the same first RCT recognition result and the second RCT recognition result; wherein the label includes RCT documents and non-RCT documents.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images