A clinical trial protocol intelligent auxiliary review method and system

Abstract

The present application relates to a kind of clinical trial scheme intelligent auxiliary review method and system, specifically includes: obtaining to be reviewed document, the structure analysis is carried out to the to-be-reviewed document, according to the analysis result from document content extraction key information, to form structured data set;According to the structured data set, review task is divided into multiple deterministic subtasks and non-deterministic subtasks;The deterministic subtask is carried out rule matching, and matching result is generated;The non-deterministic subtask is carried out semantic reasoning, and reasoning result is generated;The matching result and reasoning result of each subtask are fused, and the fusion result set is obtained;According to the fusion result set and predetermined review index, the comprehensive score of the to-be-reviewed document is calculated, and review conclusion is generated.The present application can solve the problems of low review efficiency, insufficient professional depth and unreliable results in the existing clinical trial scheme review scheme.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence technology, and in particular to an intelligent assisted review method and system for clinical trial protocols. Background Technology

[0002] Clinical trial protocols are the core guiding documents that direct the entire clinical trial process. Their accuracy, completeness, scientific validity, and compliance directly affect trial quality, participant safety, and the success or failure of final drug registration. Currently, clinical trial protocol review primarily relies on manual review, rule-based automated tools, and review systems based on a single, generalized model. However, all three methods have significant shortcomings.

[0003] Manual review, conducted by a team of clinical, statistical, and regulatory experts, is often inefficient and costly. A complex plan can take several days to several weeks to complete, and human resource costs increase linearly with the volume of reviews. Furthermore, the review results are highly dependent on the individual experience and knowledge background of the experts, making them subjective. Different experts often have different understandings of the same clause, and fatigue or subjective judgment can easily lead to errors. In the face of a large number of complex regulations and medical guidelines, there is also a risk of missing key regulations.

[0004] Automated mini-models based on rule units check the format and completeness of the solution by using preset keywords, logical rules and templates. However, these mini-models can only perform surface text matching, lack semantic understanding, cannot determine whether the content truly meets the latest specifications, have weak generalization ability, are difficult to handle qualitative tasks with ambiguous rules, and require continuous manual modification of a large number of rules as regulations and guidelines are constantly updated, resulting in high maintenance costs and slow response.

[0005] While a review system based on a single, general-purpose model can read and summarize proposals, a single model cannot simultaneously master highly specialized fields such as clinical medicine, statistics, and pharmaceutical regulations, resulting in reviews that lack depth and are superficial. Faced with complex review processes involving multiple steps such as information extraction, cross-validation, database queries, and risk assessment, this review method cannot autonomously and systematically arrange and execute these tasks. In addition, the review process often requires consulting literature, registered trial data, or performing sample size estimation and verification, and a single model lacks the ability to reliably call upon these external tools, greatly limiting its practicality.

[0006] Therefore, it is evident that the challenge lies in combining the semantic understanding capabilities of large models with the efficiency and accuracy of lightweight small models to construct an auditing system that is both capable of in-depth analysis and accurate and reliable. This system should be used to address the problems of low auditing efficiency, insufficient professional depth, and unreliable results in existing auditing technologies. Summary of the Invention

[0007] Regarding how to construct a clinical trial protocol review system that combines in-depth analysis with accuracy and reliability to solve the problems of low review efficiency, insufficient professional depth, and unreliable results in existing technologies, this invention provides an intelligent auxiliary review method and system for clinical trial protocols. By deeply integrating a lightweight processing small model for performing deterministic sub-tasks with a large language model for performing non-deterministic sub-tasks, the invention achieves automated, intelligent, and continuously optimized review of clinical trial protocols.

[0008] Firstly, according to the design scheme provided by this invention, a method for intelligent auxiliary review of clinical trial protocols is provided, comprising the following:

[0009] Obtain the document to be reviewed, perform structural analysis on the document, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set.

[0010] Based on the structured data set, the review task is divided into multiple deterministic and non-deterministic sub-tasks; based on the pre-built review knowledge base, the deterministic sub-tasks are assigned to a preset first model for rule matching to generate matching results; the non-deterministic sub-tasks are assigned to a preset second model for semantic reasoning to generate reasoning results; then, the matching results and reasoning results of each sub-task are fused to obtain a fused result set.

[0011] Based on the fusion result set and the predetermined review indicators, the comprehensive score of the document to be reviewed is calculated, and a review conclusion is generated.

[0012] Furthermore, structural analysis is performed on the document to be reviewed. Based on the analysis results, key information is extracted from the document content. The key information is then cleaned and standardized to form a structured data set, specifically including:

[0013] The hierarchical structure of the document is analyzed using a layout analysis model. Based on the analysis results, natural language processing technology is used to extract key information from the document content of each level, and all key information is cleaned and standardized. The key information corresponding to each level is integrated to form a structured data set.

[0014] Furthermore, the audit knowledge base specifically includes a rule base, a knowledge base, and an external knowledge retrieval interface:

[0015] The rule base is used to store deterministic review rules for the first model; the knowledge base includes a vector database for storing unstructured review knowledge to support the retrieval enhancement generation of the second model; the external knowledge retrieval interface is used to encapsulate standardized access interfaces to various external dynamic data sources to obtain the latest external evidence.

[0016] Further, the deterministic subtask is assigned to a preset first model for rule matching to generate matching results, specifically including:

[0017] The first model is used to call the content in the audit knowledge base and perform fast matching and logical operations with the deterministic subtask to obtain the matching result.

[0018] Furthermore, assigning the nondeterministic subtask to a preset second model for semantic reasoning and generating reasoning results specifically includes:

[0019] The second model is used to perform semantic reasoning on the nondeterministic subtask, and calls the audit knowledge base as needed to generate the reasoning result.

[0020] Furthermore, the matching results and inference results of each subtask are fused together to obtain a fusion result set, which specifically includes:

[0021] The matching results and inference results of each subtask are fused and judged. Conflicting information between the two results and information with confidence levels below the threshold in each result are marked to form a fusion result set.

[0022] Furthermore, calculating the overall score of the document to be reviewed and generating the review conclusion specifically includes:

[0023] Based on the fusion result set, the comprehensive score of the document to be reviewed is calculated according to the weight allocation logic recorded in the review indicators, and a review conclusion is generated based on the comprehensive score.

[0024] Furthermore, it also includes correcting the review conclusions, specifically including:

[0025] When an objection instruction containing revision suggestions is received regarding the current review conclusion, the current review conclusion is updated using the revision suggestions, and the updated current review conclusion is stored in the review knowledge base; when an instruction without objection is received, the current review conclusion is directly output.

[0026] Secondly, the present invention also provides an intelligent auxiliary review system for clinical trial protocols, comprising:

[0027] Document parsing and structuring module: used to obtain the document to be reviewed, perform structural analysis on the document to be reviewed, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set;

[0028] Collaborative intelligent review module: Based on the structured data set, the review task is divided into multiple deterministic and non-deterministic sub-tasks; based on a pre-built review knowledge base, the deterministic sub-tasks are assigned to a preset first model for rule matching to generate matching results; the non-deterministic sub-tasks are assigned to a preset second model for semantic reasoning to generate reasoning results; then, the matching results and reasoning results of each sub-task are fused to obtain a fusion result set.

[0029] The review result generation module is used to calculate the comprehensive score of the document to be reviewed based on the fusion result set and the predetermined review indicators, and to generate the review conclusion.

[0030] Thirdly, the present invention also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, cause the processor to perform the method as described in the first aspect.

[0031] The beneficial effects of this invention are:

[0032] (1) By dividing the review task into deterministic sub-tasks and non-deterministic sub-tasks, and assigning them to the first model for performing deterministic sub-tasks and the second model for performing non-deterministic sub-tasks respectively for review, the first model can be used to achieve efficient and accurate matching of rule-based tasks, and the semantic understanding ability of the second model can be called to cover complex scenarios that require deep reasoning, making the review results more accurate and reliable.

[0033] (2) The high-frequency and simple deterministic subtasks are assigned to the first model for processing, which reduces the use of the second model's computing resources and effectively reduces costs while ensuring overall review efficiency;

[0034] (3) Through the review mechanism of the present invention, the review speed is improved, the risk of omission due to fatigue and negligence in pure manual review is effectively overcome, as well as the subjective bias caused by different expert standards, thereby improving the quality and consistency of review. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an intelligent assisted review method for clinical trial protocols in an embodiment of the present invention.

[0036] Figure 2This is a schematic diagram of the architecture of an intelligent auxiliary review system for clinical trial protocols in an embodiment of the present invention; Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0038] Example 1

[0039] like Figure 1 As shown, Embodiment 1 of the present invention provides an intelligent assisted review method for clinical trial protocols, comprising:

[0040] S101. Obtain the document to be reviewed, perform structural analysis on the document to be reviewed, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set.

[0041] Furthermore, structural analysis is performed on the document to be reviewed. Based on the analysis results, key information is extracted from the document content. The key information is then cleaned and standardized to form a structured data set, specifically including:

[0042] The hierarchical structure of the document is analyzed using a layout analysis model. Based on the analysis results, natural language processing technology is used to extract key information from the document content of each level, and all key information is cleaned and standardized. The key information corresponding to each level is integrated to form a structured data set.

[0043] Specifically, the hierarchical structure includes headings, paragraphs, tables, and signature areas. Based on the analysis results, natural language processing technology is used to extract heterogeneous information (key information) such as basic project information (e.g., program name, principal investigators) and key review information (e.g., inclusion / exclusion criteria, research design type, intervention measures, statistical methods, etc.) to achieve the structured transformation of unstructured text. The extracted heterogeneous information undergoes format standardization processing, such as unifying dates in different formats to a standard format, standardizing monetary units, and mapping research types in free text to preset classification codes, to eliminate the interference of data heterogeneity on subsequent automated review and form a structured dataset.

[0044] S102. Based on the structured data set, the review task is divided into multiple deterministic sub-tasks and non-deterministic sub-tasks; based on the pre-built review knowledge base, the deterministic sub-tasks are assigned to a preset first model for rule matching to generate matching results; the non-deterministic sub-tasks are assigned to a preset large language model for semantic reasoning to generate reasoning results; then, the matching results and reasoning results of each sub-task are fused to obtain a fusion result set.

[0045] Furthermore, the audit knowledge base specifically includes a rule base, a knowledge base, and an external knowledge retrieval interface:

[0046] The rule base stores configurable, versioned deterministic review rules for a dedicated first model. These rules can be structured and compiled by experts according to regulatory guidelines (including the "Guidelines for the Establishment of Ethics Review Committees for Clinical Research Involving Human Subjects") and internal standard operating procedures, and can cover scenarios that require precise logical judgment, such as formal review, risk classification, and type verification.

[0047] The knowledge base, centered on a vector database, stores unstructured deep semantic knowledge to support the retrieval enhancement generation of the second model. Its content includes, but is not limited to: domestic and international drug regulatory regulations and ethical guidelines, core journal articles and treatment standards in the field, anonymized historical review cases, and expert feedback.

[0048] The review knowledge base is also linked to various external dynamic data sources (including PubMed, ClinicalTrials.gov, the Chinese Clinical Trial Registry, etc.), enabling the system to obtain the latest external evidence.

[0049] Specifically, a task planner and scheduler is used to decompose the overall review task into multiple atomic subtasks with dependencies, based on a built-in review process model. The scheduler intelligently distributes the atomic subtasks according to their nature: deterministic subtasks with clear rules and requiring precise execution (such as formal review, risk level determination, and sample size formula verification) are assigned to the first model; non-deterministic subtasks requiring deep semantic understanding and logical reasoning (such as the innovativeness assessment of the research background and the scientific validity judgment of the research design) are assigned to the large language model.

[0050] Further, the deterministic subtask is assigned to a preset first model for rule matching to generate matching results, specifically including:

[0051] The first model is used to call the corresponding rule set in the rule base, and to perform fast matching and logical operations between the rule set and the deterministic subtask to obtain the matching result.

[0052] A first model is used to perform deterministic review tasks. The first model is used to receive instructions from the scheduler, load the corresponding rule set from the rule base, apply it to the input deterministic sub-task, perform fast matching and logical operations through the rule engine, and output structured verification results (such as "pass / fail", the specific rule clause number violated, and the calculated risk level).

[0053] Furthermore, assigning the nondeterministic subtask to a preset second model for semantic reasoning and generating reasoning results specifically includes:

[0054] The second model is used to perform semantic reasoning on the nondeterministic subtask, and calls the audit knowledge base as needed to generate the reasoning result.

[0055] Among them, a large language model enhanced with medical knowledge is used to handle nondeterministic subtasks requiring complex cognitive abilities. Upon receiving task instructions and solution context, it can autonomously plan reasoning paths and proactively invoke external tools as needed. For example, it can retrieve relevant guidelines and literature from a knowledge base using retrieval enhancement technology, or query the latest clinical evidence through an external knowledge retrieval module to perform chained reasoning, ultimately generating review opinions and scores with supporting evidence and confidence levels.

[0056] Furthermore, the matching results and inference results of each subtask are fused together to obtain a fusion result set, which specifically includes:

[0057] The matching results and inference results of each subtask are fused and judged. Conflicting information between the two results and information with confidence levels below the threshold in each result are marked to form a fusion result set.

[0058] Specifically, the interaction of different model units is coordinated, and a comprehensive judgment is made on the multi-source outputs. For specific complex audit items, a preset collaborative strategy is activated. For example, after the large language model identifies potential risk points, the small model is triggered to perform precise matching verification in the historical case library. At the same time, the output results of all sub-tasks are merged, and information that may conflict or has low confidence is marked to form a unified and structured fusion result set.

[0059] S103. Based on the fusion result set and the predetermined review indicators, calculate the comprehensive score of the document to be reviewed and generate a review conclusion.

[0060] Furthermore, calculating the overall score of the document to be reviewed and generating the review conclusion specifically includes:

[0061] Based on the fusion result set, the comprehensive score of the scheme is calculated according to the preset review indicators and their weight ratios. Then, based on the total score, whether there are any major defects (such as the existence of similar studies), and the confidence level of each sub-task, a preliminary review conclusion (such as "agree", "agree after modification") is automatically generated according to the built-in judgment logic.

[0062] The system automatically populates structured data such as basic project information, risk classification results, detailed scores for each dimension, and review conclusions into a pre-set review report template. Simultaneously, it utilizes a large language model to integrate and refine scattered review comments, generating a coherent and professional review report with detailed review comments.

[0063] Furthermore, it includes an interactive platform, featuring a visual dashboard and report details page that clearly displays the review progress, overall score, risk distribution, and the basis for each review item. It supports clicking on highlighted risk points to quickly locate them in the original text and provides links to traceable evidence. By showing experts the preliminary review report generated by the system, experts can review any review item and modify scores and comments or add annotations. The system highlights review items with low confidence levels to guide experts to focus on them.

[0064] Furthermore, it also includes correcting the review conclusions through a pre-set feedback module:

[0065] Regarding the current review conclusion, when the feedback module receives an objection containing a correction suggestion, it updates the current review conclusion using the correction suggestion and stores the current review conclusion in the audit knowledge base; when the feedback module receives a no-objection instruction, it outputs the current review conclusion.

[0066] Specifically, when experts disagree with the system's judgment and make corrections, their revised data can be captured by the system. This processed data will be used to periodically update the rule base or as high-quality training data to fine-tune and optimize large language models, thus enabling the system to continuously learn and evolve. After the experts confirm the data is correct, the system generates a final review report with an electronic signature and archives it.

[0067] The present invention provides an intelligent assisted review method for clinical trial protocols. By dividing the review task into deterministic sub-tasks and non-deterministic sub-tasks, and assigning them to a first model and a second model for review respectively, it not only ensures the accurate execution of tasks with clear rules, but also covers complex scenarios that require semantic understanding, making the review results more accurate and reliable.

[0068] Example 2

[0069] This embodiment 2 provides an intelligent assisted review method for clinical trial protocols, the specific method of which is as follows:

[0070] S201. Obtain the document to be reviewed, perform structural analysis on the document to be reviewed, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set.

[0071] Specifically, users first upload clinical trial protocol documents. If the file format does not meet the requirements, they need to re-upload it. The document is then categorized and recognized according to its format. If the document is in image format, it is directly sent to the OCR (Optical Character Recognition) unit; if it is a PDF, the text layer is extracted first, and if there is no text layer, it is rendered as an image before being put into the OCR unit.

[0072] Specifically, the document structure is identified through a layout analysis model, outputting the type of each area (title, body text, tables, etc.) and their coordinates. A named entity recognition model extracts predefined fields from the text in each area and records their confidence levels. Fields with low confidence (confidence < 0.8) are marked as "awaiting manual confirmation" in the structured data. A format standardization module unifies the format of the extracted fields, such as date, amount, and code mapping. The output is structured and standardized data for subsequent steps.

[0073] S202. Based on the structured data set, the review task is divided into multiple deterministic sub-tasks and non-deterministic sub-tasks; based on the review knowledge base, the deterministic sub-tasks are assigned to a preset first model for rule matching to generate matching results; the non-deterministic sub-tasks are assigned to a preset second model for semantic reasoning to generate reasoning results; then, the matching results and reasoning results of each sub-task are fused to obtain a fusion result set.

[0074] Specifically, based on the review process template, the review tasks are broken down into quantitative (deterministic) and qualitative (non-deterministic) sub-tasks. The quantitative tasks include formal review, risk classification assessment, sample size estimation verification, data consistency verification (such as the consistency between registration information and the protocol), and statistical method compliance check. The qualitative tasks include evaluation of the research background and justification, judgment of the rationality of the research design, evaluation of the appropriateness of the outcome indicators, evaluation of the rationality of the inclusion and exclusion criteria, evaluation of the completeness of the research process, evaluation of the rationality of data management, evaluation of risk management measures, and evaluation of innovation.

[0075] It also includes a task planner and a scheduler. The task planner is used to construct a directed acyclic graph (DAG) based on the dependencies between tasks (e.g., risk grading needs to be completed before innovation assessment because the risk level affects the weight of the innovation judgment). The scheduler is used to start executing the DAG, first executing tasks without dependencies (e.g., formal review, risk grading), and then triggering subsequent tasks upon completion.

[0076] The scheduler sends quantitative tasks to small model execution units and qualitative tasks to large model execution units. Small model execution units process multiple quantitative tasks in parallel. Each task's processing flow includes: first, loading the corresponding rule set from the rule base (formal review rule sets include version number checks, page number checks, informed consent statement checks, etc.); then, extracting the required fields from the quantitative task as rule inputs; finally, the rule unit performs matching, outputs the results, and caches the results. After all quantitative tasks are completed, the results are aggregated to the cross-validation and fusion unit.

[0077] Due to the long inference time of large models, asynchronous processing is adopted. Each task is placed in a message queue and executed by multiple large models. The processing flow of each qualitative task includes: according to the task type, the large model starts inference. During the inference process, if a retrieval is required, a retrieval call is triggered in the review knowledge base. For example, if the large model finds that the solution mentions "there are currently no drugs targeting XX", it can actively call the RAG retrieval to search the knowledge base for whether there are any marketed drugs or clinical trials targeting that target. After the retrieval results are returned, the large model continues inference and finally outputs the score, evidence list, and confidence level. After all qualitative tasks are completed, the results are summarized in the cross-validation and fusion unit. The first model includes lightweight small models, and the second model includes a large language model.

[0078] The cross-validation and fusion unit receives the results of all subtasks. For tasks involving both large and small models, the following processing is performed: The results of the small model are retrieved from a list of similar studies in the internal historical case database; the results of the large model are retrieved from an external registry of similar studies. The two lists are merged, duplicates are removed, and a final list is generated. If the list is not empty, it is marked "Similar studies exist," and all study IDs are listed. For other tasks, the fusion unit only summarizes the results; cross-validation is not required. The complete fused result set (containing scores, justifications, evidence, risk levels, etc. for all subtasks) is the fused result set.

[0079] S203. Based on the fusion result set and the predetermined review indicators, calculate the comprehensive score of the document to be reviewed and generate a review conclusion.

[0080] The review result generation module receives the fusion result set and performs a weighted summation according to the weights of each indicator. For example: scientific validity 50%, standardization 20%, feasibility 15%, and innovation 15%. The score for each dimension is obtained by multiplying the average score of all sub-tasks under that dimension by its weight. Based on the total score, whether there is similar research, and whether there are any modification suggestions (e.g., when the confidence level is <0.8 or manually marked), the review conclusion is automatically determined and detailed scores for each dimension are generated.

[0081] Next, the structured information is filled into the report template, including basic project information, risk classification, scores for each dimension, review conclusions, etc.; and the review comments for each qualitative task are integrated and polished to generate a fluent "Specific Review Comments" text, along with links to all evidence.

[0082] The system calls a PDF generation library to render the report content into a PDF file, adds watermarks and page numbers, outputs a preliminary review report, stores it in a temporary directory, and pushes it to the human-machine collaboration business interface.

[0083] The human-machine collaborative interface is used by experts to log in to the system, enter the list of proposals to be reviewed, click on the proposal, and view the review details page. The page displays the preliminary review report, with the original proposal text on the left and the report content on the right. Experts can check each item one by one. For scores or comments, experts can click the "edit" button to open an editing box and make modifications. For low-confidence items (highlighted in yellow), the system suggests that experts focus on reviewing them. Experts can click on the evidence link to jump to an external database to view the original text. Experts can add annotations, such as "It is recommended to supplement the basis for the sample size calculation."

[0084] After the expert makes revisions, the system recalculates the total score and conclusion in real time and updates the report preview. If the expert believes the system's judgment is incorrect, they can click the "Correction" button, select the error type, and fill in the correct information. This feedback data is recorded in the feedback database. After the expert completes the review, they click "Confirm and Generate Final Report." The system saves the final report (including all expert revisions) to the official storage and triggers subsequent processes (such as email notifications and archiving).

[0085] The system generates a final PDF report with an electronic signature and stores the report content in a database for easy retrieval and auditing later.

[0086] After archiving is complete, the system asynchronously initiates a model optimization task: compiling the misjudged cases reported by experts into training data. For small model rule bases, the system analyzes the misjudged cases; if the misjudgment is due to insufficient rule coverage, the system prompts the rule administrator to update the rules. For large models, the expert-corrected comments are used as high-quality samples for subsequent prompt word optimization or model fine-tuning. The optimized model and rules take effect in the next review, forming a continuous improvement loop.

[0087] By learning from the expert-modified review conclusions through the method of this embodiment, the review trends can be updated in a timely manner, making the review conclusions more in line with reality. High-frequency, simple deterministic subtasks are assigned to lightweight small models, reducing the use of large language model computing resources, thus reducing costs while ensuring overall efficiency.

[0088] Example 3

[0089] This embodiment provides an intelligent auxiliary review system for clinical trial protocols, and the specific implementation and interaction of its modules are as follows:

[0090] Document parsing and structuring module: used to obtain the document to be reviewed, perform structural analysis on the document to be reviewed, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set;

[0091] The specific processing steps of the document parsing and structuring module include:

[0092] First, users upload clinical trial protocol documents through the human-machine collaboration business interface;

[0093] Next, the clinical trial protocol document undergoes layout analysis, OCR recognition, information extraction, and format standardization. The layout analysis process includes: using a visual-language model to analyze the document's layout structure, identifying titles, paragraphs, tables, images, etc., and outputting the coordinates and type of each area. The OCR recognition process includes: performing OCR recognition on scanned documents or images, recognizing text in non-text areas, and outputting confidence scores. The information extraction process includes: using a BERT-based named entity recognition model to extract predefined fields from the text, including basic project information and key review information.

[0094] The basic project information includes the protocol name, protocol number, principal investigator, initiating institution, funder, funding content, research type, and intervention type; the key review information includes the inclusion criteria, exclusion criteria, research design type, randomization method, blinding, sample size, statistical methods, research flowchart description, and adverse event handling; the standardization process includes standardizing the extracted fields, for example, standardizing the format of dates, amounts, and codes.

[0095] Finally, the output includes information such as field names, field values, source page numbers, and confidence levels. The document parsing and structuring module is primarily based on a small model.

[0096] Collaborative intelligent review module: Based on the structured data set, the review task is divided into multiple deterministic and non-deterministic sub-tasks; based on a pre-built review knowledge base, the deterministic sub-tasks are assigned to a preset first model for rule matching to generate matching results; the non-deterministic sub-tasks are assigned to a preset second model for semantic reasoning to generate reasoning results; then, the matching results and reasoning results of each sub-task are fused to obtain a fusion result set.

[0097] The audit knowledge base unit specifically includes a rule base, a knowledge base, and an external knowledge retrieval interface;

[0098] The rule base used by the first model employs a rule engine to store deterministic rules and supports versioned management. Each rule can include: rule ID, name, description, effective date, expiration date, condition (e.g., learning type is interventional research), and action (e.g., risk level is high risk).

[0099] The knowledge base for the second model uses a vector database to store unstructured knowledge. The specific stored content includes: regulatory guidelines documents segmented by paragraph and indexed by vector, text fragments of treatment guidelines (including NCCN guidelines and CSCO guidelines), and historical review cases (including historical protocols and their review opinions and expert feedback).

[0100] The external knowledge retrieval interface is used to encapsulate API calls to multiple external databases, including calls to websites such as PubMed E-utilities, ClinicalTrials.gov API, ChiCTR, and UpToDate.

[0101] like Figure 2 As shown, the collaborative intelligent auditing module includes a task planner and scheduler, a small model execution unit, a large model execution unit, and a cross-validation and fusion unit;

[0102] The task planner and scheduler receives structured solution data and dynamically generates a task list based on a preset review process template. Each task includes: task ID, type (quantitative / qualitative), dependencies, invoked rules / knowledge base, and timeout settings. In terms of task priority, preliminary tasks such as formal review and risk assessment have the highest priority, triggering subsequent tasks only after their completion. The scheduling strategy uses a directed acyclic graph (DAG) approach to manage task dependencies, concurrently executing tasks without dependencies.

[0103] The small model can be a rule-based model such as PubMedBERT or BioBERT, adapted for medical texts and specific review tasks. The small model execution unit receives instructions for quantitative tasks (deterministic subtasks), loads the corresponding rule set from the rule base, performs rule matching on the input structured data, and outputs the results. It can output results for formal review tasks and risk grading tasks. All matching results output by the small model also include an execution timestamp and rule version number.

[0104] The large model execution unit receives instructions for qualitative tasks (non-deterministic subtasks) and performs large model inference. The large model uses open-source large language models such as LLaMA-3-70B or ChatGLM-6B, fine-tuned for the medical field, and is deployed on a GPU cluster. The specific process includes: input to each qualitative task includes a task description, relevant text sections of the solution, and a list of tools to be called (such as RAG retrieval and external retrieval); the large model can actively trigger RAG retrieval and call external retrieval interfaces during inference. Output results include a score, reasoning, cited evidence links, and confidence level. The first model includes a rule model, and the second model includes a large language model.

[0105] The cross-validation and fusion unit is used to receive the outputs of both the small and large models and compare them for tasks simultaneously assigned to both models. The fusion strategy includes: if the small model finds identical studies in its internal case library and the large model finds similar studies in an external registry, the fusion output is "Similar studies exist," listing all study IDs; if the results conflict (e.g., the small model has no results, but the large model has results), it is marked as "Requires manual confirmation" and highlighted. The fused result serves as the final output of the task.

[0106] By using a rule model and a large language model for joint review, we can not only achieve efficient and accurate matching of rule-based tasks using the rule model, but also use the semantic understanding capabilities of the large language model to cover complex scenarios that require deep reasoning, making the review results more accurate and reliable.

[0107] It may also include a risk stratification assessment module led by a small model, which is essentially a dedicated, independently running small model. Its input consists of risk-related fields extracted from structured data, such as study type, intervention type, whether it is off-label, whether it involves vulnerable populations, and whether it involves invasive procedures. Its output includes: risk level, risk label (low / medium / high / very high), and review format (simplified / conference). This result is stored in structured data for later use.

[0108] The review result generation module is used to calculate the comprehensive score of the document to be reviewed based on the fusion result set and the predetermined review indicators, and to generate the review conclusion.

[0109] The audit results generation module includes comprehensive scoring and rating determination, and report generation.

[0110] The comprehensive scoring and grading process includes summarizing the scores of all sub-tasks and calculating the weighted average of each level of indicators to calculate the total score, with a maximum score of 100.

[0111] Review conclusion determination rules: If the result of the "similar research" task is "there are completed or ongoing similar studies", then it is directly determined as "disagree"; otherwise, if the total score is ≥60 and there are no modification comments (i.e., the confidence level of all qualitative tasks is >0.8 and there are no manual labels), then it is determined as "agree"; if the total score is ≥60 but there are modification comments (such as confidence level <0.8 or manual labels), then it is determined as "agree after modification"; if the total score is <60 and there are modification comments, then it is determined as "review after modification"; the final output includes the total score, review conclusion, and detailed scores for each dimension.

[0112] The report generation process includes: using a small model to input basic project information, risk classification results, scores for each dimension, and review conclusions into the corresponding fields of the template; using a large model to integrate the review comments from each qualitative task, generating a coherent "Specific Review Comments" text, along with links to all cited evidence; and finally, outputting a complete preliminary review report.

[0113] It also includes a human-machine collaborative business interface, which can be configured as follows: This human-machine collaborative review interface adopts a responsive layout and can be adapted to both PC and mobile devices. After entering the system, the home dashboard will centrally display the list of solutions to be reviewed, the statistics of completed reviews, and an overview of risk distribution. Clicking on any solution will take you to the details page, which adopts a three-column layout: the original PDF is displayed on the left, the structured data summary is in the middle, and the preliminary review report automatically generated by the system is on the right. Each review comment in the report is associated with the position of the original PDF on the left, and clicking on it will allow you to pinpoint the exact location. All risk points are visually marked with red, yellow, and green lights, and clicking on them will allow you to view the basis for the judgment. Each qualitative review task will display a confidence level, and items with a confidence level below 0.8 will be automatically highlighted with a yellow background to remind human attention. During expert review, scores, comments, or annotations can be directly modified, and the system will update the total score and conclusion in real time. There is an error correction button below each review item. After clicking, you can select the error type (such as rule misjudgment, insufficient evidence, or others) and fill in the correct information. These error correction records will be automatically stored in the feedback database for subsequent model iteration and optimization. Once the experts confirm that everything is correct, they click to generate the final report. The system then generates an official PDF with an electronic signature and automatically archives it, while simultaneously notifying the relevant parties via email.

[0114] By collecting expert feedback through a human-computer interaction interface and inputting the feedback results into the model for comprehensive application, the system can continuously adapt to new review requirements and standards, achieving continuous optimization. Furthermore, this business interface design provides intuitive risk visualization, accurate original text location, and convenient review tools, allowing experts to focus on high-value decisions and maximizing the efficiency of human-machine collaboration.

[0115] The present invention provides an intelligent auxiliary review system for clinical trial protocols. By integrating a large language model with a rule model, and setting up a review knowledge base unit, a document parsing and structuring module, a task planning and scheduler, a cross-validation and fusion unit, and a result generation module, the system achieves both accurate and rapid review conclusions through the combined action of these modules. Furthermore, it delegates high-frequency and simple quantitative tasks to the rule model, significantly reducing the computational resources required by the large language model and optimizing costs while ensuring overall efficiency.

[0116] It should be noted that the intelligent auxiliary review system for clinical trial protocols provided in this embodiment of the invention can be specifically referred to in the above method embodiments to achieve the functions described above.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent assisted review of clinical trial protocols, characterized in that, Include: Obtain the document to be reviewed, perform structural analysis on the document, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set. Based on the structured data set, the review task is divided into multiple deterministic sub-tasks and non-deterministic sub-tasks; Based on a pre-built audit knowledge base, the deterministic sub-task is assigned to a preset first model for rule matching to generate matching results; The nondeterministic subtasks are assigned to a preset second model for semantic reasoning to generate reasoning results; then, the matching results of each subtask are fused with the reasoning results to obtain a fused result set. Based on the fusion result set and the predetermined review indicators, the comprehensive score of the document to be reviewed is calculated, and a review conclusion is generated.

2. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, The document to be reviewed undergoes structural analysis. Based on the analysis results, key information is extracted from the document content. This key information is then cleaned and standardized to form a structured data set. Specifically, this includes: The hierarchical structure of the document is analyzed using a layout analysis model. Based on the analysis results, natural language processing technology is used to extract key information from the document content of each level, and all key information is cleaned and standardized. The key information corresponding to each level is integrated to form a structured data set.

3. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, The audit knowledge base specifically includes a rule base, a knowledge base, and an external knowledge retrieval interface: The rule base is used to store deterministic review rules for the first model; the knowledge base includes a vector database for storing unstructured review knowledge to support the retrieval enhancement generation of the second model; the external knowledge retrieval interface is used to encapsulate standardized access interfaces to various external dynamic data sources to obtain the latest external evidence.

4. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, The deterministic subtask is assigned to a preset first model for rule matching to generate matching results, specifically including: The first model is used to call the content in the audit knowledge base and perform fast matching and logical operations with the deterministic subtask to obtain the matching result.

5. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, The nondeterministic subtask is assigned to a pre-defined second model for semantic reasoning, and the reasoning results are generated specifically including: The second model is used to perform semantic reasoning on the nondeterministic subtask, and calls the audit knowledge base as needed to generate the reasoning result.

6. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, The matching results of each subtask are fused with the inference results to obtain a fusion result set, which specifically includes: The matching results and inference results of each subtask are fused and judged. Conflicting information between the two results and information with confidence levels below the threshold in each result are marked to form a fusion result set.

7. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, Calculating the overall score of the document to be reviewed and generating the review conclusion specifically includes: Based on the fusion result set, the comprehensive score of the document to be reviewed is calculated according to the weight allocation logic recorded in the review indicators, and a review conclusion is generated based on the comprehensive score.

8. The intelligent assisted review method for clinical trial protocols according to claim 1, characterized in that, It also includes correcting the review conclusions, specifically including: When an objection instruction containing revision suggestions is received regarding the current review conclusion, the current review conclusion is updated using the revision suggestions, and the updated current review conclusion is stored in the review knowledge base; when an instruction without objection is received, the current review conclusion is directly output.

9. A clinical trial protocol intelligent auxiliary review system, characterized in that, Include: Document parsing and structuring module: used to obtain the document to be reviewed, perform structural analysis on the document to be reviewed, extract key information from the document content based on the analysis results, and perform data cleaning and standardization on the key information to form a structured data set; Collaborative intelligent review module: used to divide the review task into multiple deterministic sub-tasks and non-deterministic sub-tasks based on the structured data set; Based on a pre-built audit knowledge base, the deterministic sub-task is assigned to a preset first model for rule matching to generate matching results; The nondeterministic subtasks are assigned to a preset second model for semantic reasoning to generate reasoning results; then, the matching results of each subtask are fused with the reasoning results to obtain a fused result set. The review result generation module is used to calculate the comprehensive score of the document to be reviewed based on the fusion result set and the predetermined review indicators, and to generate the review conclusion.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the method as described in claims 1 to 8.

Images