Adverse event report quality control method and system based on rule engine and large model cooperation

Abstract

The application provides an adverse event report quality control method and system of a rule engine cooperating with a large model, and belongs to the technical field of medical supervision based on natural language processing; first, multi-source system original report data is acquired, preprocessed into adverse event report data, and parsed into a structured field set and an unstructured text set, a cross verification mapping relationship between the two is established, and a quality control rule set is constructed; a rule engine quality control module is designed, a rule engine quality control result is output, a large language model quality control module is designed, and a large model quality control result is generated; finally, a collaborative decision module is designed, a collaborative decision result is obtained by weighted fusion according to the rule engine quality control result and the large model quality control result, and the difference items between the two are analyzed, and a difference detection result is calculated. The application improves the comprehensiveness, accuracy and explainability of quality control, thereby better meeting the compliance audit and data governance needs in the adverse event monitoring field.

Description

Technical Field

[0001] This invention belongs to the field of medical supervision technology based on natural language processing, and in particular relates to a method and system for quality control of adverse event reporting that combines a rule engine and a large model. Background Technology

[0002] With the continuous advancement of adverse event monitoring for medical devices and pharmaceuticals, the quality of reported data has become a crucial factor influencing risk signal identification, adverse event cause analysis, and regulatory decision-making. Adverse event reports are characterized by a large number of fields, the coexistence of structured and unstructured information, and complex business constraints. Issues such as missing key information, non-standard terminology, inconsistencies between fields, non-compliance with regulatory requirements, or abnormal time logic will directly impact the effective conduct of subsequent monitoring, analysis, and re-evaluation. Furthermore, the cumulative effect of reported data quality defects across various adverse event monitoring systems is intensifying, posing a serious threat to patient safety and public health decision-making.

[0003] Currently, the main methods for quality control of adverse event reporting include the following: (1) Rule Engine-Based Report Quality Control Method: This method mainly uses rule engine technologies such as Drools, Flink CEP, or Python rule-engine to encode quality control requirements such as integrity verification, format verification, enumeration value verification, and cross-field logic verification into a structured rule set. The report data is automatically reviewed through the rule sequence matching and triggering mechanism. This type of method has the advantages of strong rule execution determinism and transparent and auditable reasoning process. However, this method usually relies on domain experts to manually write and maintain rules. The rule coverage is limited by the expert's experience boundary, making it difficult to effectively handle semantic ambiguity, terminology diversity, and implicit clinical logical connections in event statement texts, thus limiting the applicability of this method in unstructured quality control scenarios. (2) Report analysis methods based on deep learning and large language models: These methods utilize pre-trained language models such as BERT, DeepSeek, and GPT to perform named entity recognition, text classification, and semantic matching on report texts, automatically extracting key quality control information from unstructured event descriptions. These methods have the advantages of strong semantic understanding and no need for manually defined rules. However, due to the black-box nature of the reasoning process of deep learning models, their decision-making basis is difficult to trace and audit, failing to meet the compliance audit requirements of at least 80% interpretability in the field of adverse event monitoring for medical devices. Furthermore, large models have significant performance deficiencies in numerical precision calculation, temporal logic verification, and deterministic constraint checks, and the generated quality control conclusions are prone to illusion problems. Relying solely on large models is insufficient to meet the accuracy standards for regulatory compliance. (3) Adverse event analysis methods based on statistical signal detection: This method uses statistical methods such as Proportional Reporting Ratio (PRR), Reporting Odds Ratio (ROR), and Empirical Bayes Geometric Mean (EBGM) to identify abnormal signals by comparing the deviation between the observed number of reports and the expected number of reports. This type of method has played an important role in signal mining of large-scale adverse event databases. However, this type of method relies on the completeness and accuracy of the report data as input premises, and it does not have the ability to review the quality of reports. It is difficult to accurately locate and classify the content defects of a single report, and it lacks a mechanism for comprehensively evaluating the quality of report data from the dimensions of information completeness, standardization of filling, and consistency of fields.

[0004] In summary, existing quality control methods for adverse event reporting have certain limitations in three dimensions: deterministic verification capability, semantic understanding depth, and compliance interpretability. Rule engine methods have limited rule coverage and cannot handle unstructured semantics, deep learning methods have insufficient interpretability and weak numerical logic verification capability, and statistical signal detection methods lack single-report level quality control evaluation capability and are highly sensitive to the quality of input data. Summary of the Invention

[0005] To address the aforementioned issues, this invention develops a quality control method and system for adverse event reporting that deeply integrates the deterministic quality control capabilities of rule engines with the flexible semantic reasoning capabilities of large models. This improves the comprehensiveness, accuracy, and interpretability of quality control, thereby better meeting the compliance audit and data governance needs in the field of adverse event monitoring.

[0006] The first aspect of this invention proposes a quality control method for adverse event reporting that integrates a rule engine and a large model, comprising the following processes: Acquire raw report data from multi-source systems and preprocess it into adverse event report data; Adverse event report data is parsed into a set of structured fields and a set of unstructured text, and a cross-validation mapping relationship is established between the two. At the same time, a set of quality control rules is constructed. Input the structured field set and quality control rule set into the rule engine quality control module, and output the rule engine quality control results, including the rule engine quality control result data object, the rule engine dimension quality score and the rule engine dimension confidence. The unstructured text set, quality control rule set, and cross-validation mapping relationship are input into the large language model quality control module. Prompt word generation, prompt word conversion, and confidence evaluation are performed respectively to obtain the large model quality control results, which include large model quality control result data objects, large model dimension quality scores, and large model dimension confidence. The rule engine quality control results, large model quality control results, and predefined confidence weight vectors are input into the collaborative decision-making module. The confidence weight vector corresponding to the current report type is read. The collaborative decision-making results are obtained by performing a weighted fusion of the rule engine dimension quality score and the large model dimension quality score based on the confidence of the rule engine dimension and the confidence of the large model dimension. Discrepancies between the rule engine quality control results and the large model quality control results are detected, and the discrepancy detection results are calculated.

[0007] Preferably, the step of parsing the adverse event report data into a structured field set and an unstructured text set specifically involves: First, based on the pre-configured field parsing mapping table, fields with clearly defined value range constraints, including report number, report date, report nature, product name, model and specifications, registration certificate number, product batch number, production date, expiration date, usage time, implantation date, discontinuation date, event consequences, correlation evaluation, and reporter's position, are extracted from the adverse event report data and categorized and assembled into a structured field set. Second, free text fields, including event description, preliminary cause analysis of the event, description of treatment measures, and description of the time of improvement of the adverse event, are extracted from this data and categorized and assembled into an unstructured text set.

[0008] Preferably, the construction of the quality control rule set specifically involves: First, based on the regulatory requirements for adverse event monitoring of medical devices and the clinical reporting business specifications, non-empty check rules for mandatory fields are defined from the perspective of completeness. Core mandatory fields include report number, report date, event occurrence date, discovery or awareness date, product name, model / registration certificate number, event consequences, death / serious injury details, and reporting subject. Other fields are treated as ordinary mandatory fields. From the perspective of standardization, enumerated value range matching rules and date format validation rules are defined. From the perspective of consistency, cross-field logical consistency rules are defined. From the perspective of compliance, report timeliness rules are defined. From the perspective of logic, time... The initial quality control rule set is formed by first establishing the order rules and the numerical range rules. Then, each quality control rule in the initial set is structured using JSON Schema format to obtain the final quality control rule set. Rule fields include rule identifier, dimension, applicable field name, judgment logic expression, error level, and weight coefficient. The error level distinguishes between warning, general, and severe defects, and the complete absence of core required fields is separately identified by a core required field defect marker. Finally, the quality control rule set is loaded into the rule engine's rule base, and a corresponding confidence weight coefficient is configured for each rule.

[0009] Preferably, the rule engine quality control module consists of a rule loader, a rule matcher, and a confidence calculator. First, the rule loader reads a subset of quality control rules corresponding to the current report type from the quality control rule set and compiles it into an executable matching network within the rule engine. Second, the rule matcher performs deterministic matching between each field value in the structured field set and the corresponding matching network, outputting a rule engine quality control result data object, and adding a core required field defect marker when all core required fields are completely missing. Finally, the confidence calculator calculates the rule engine dimension quality score and the rule engine dimension confidence score, and packages the rule engine quality control result data object, the rule engine dimension quality score, and the rule engine dimension confidence score into a rule engine quality control result.

[0010] Preferably, the step of using a confidence calculator to calculate the quality score and confidence score of the rule engine dimension specifically involves: First, a preset weight coefficient is read from the metadata of each rule. This weight coefficient ranges from zero to one, reflecting the relative importance of different quality control rules in the overall quality evaluation. Rules with missing core required fields have the highest weight coefficient, while rules with formatting prompts have the lowest. Second, rules that pass the check are summed according to their weight coefficients, while rules that fail are treated as zero. The weighted sum is then divided by the sum of all rule weight coefficients to obtain the dimension standardized to the zero-to-one range. The rule engine dimension quality score The closer this value is to one, the higher the pass rate of the report under the applicable rules of the rules engine; secondly, the calculation dimension Rule engine dimension confidence This confidence level is determined by the dimensions in the current report. The ratio of the number of rules that can actually be executed within a dimension to the total number of applicable rules for that dimension is determined, i.e., the dimension The rule applicability rate reflects the rule engine's applicability to dimensions. The adequacy of quality control coverage reflects the extent to which the rule engine covers the quality control of the report. The closer the value is to one, the more adequacy the quality control coverage of the report by the rule engine.

[0011] Preferably, the large language model quality control module consists of a prompt word builder, a large language model inference engine, and a semantic confidence evaluator. First, the prompt word builder transforms the quality control rule set and unstructured text set into inference prompt words for unstructured text, while simultaneously transforming the cross-validation mapping relationship into a cross-validation prompt template. Second, the large language model inference engine loads the pre-trained large language model and processes each inference prompt word, generating semantic quality control judgment results for five dimensions: completeness of event statement, terminology standardization, semantic consistency, report timeliness, and logical rationality. Based on the cross-validation prompt template, it performs a consistency comparison between structured field values ​​and unstructured text semantic elements, outputting the large model quality control result data object. Finally, the semantic confidence evaluator calculates the large model dimensional quality score and the large model dimensional confidence, packaging the large model quality control result data object, the large model dimensional quality score, and the large model dimensional confidence into the large model quality control result.

[0012] Preferably, the step of using a semantic confidence evaluator to calculate the quality score and confidence score of the large model dimension is as follows: First, calculate the quality score for the large model dimensions. For large language models in terms of dimensionality For each semantic quality control judgment result, its judgment conclusion and corresponding defect level are extracted. A quality scoring logic consistent with the rule engine is adopted: entries that pass the judgment are weighted and summed according to preset entry weight coefficients; entries that fail the judgment are treated as zero values, and the weighted sum is divided by the dimension. The sum of the weights of all decision items is used to obtain a large model dimensionality quality score standardized to the zero-to-one interval. The closer the value is to a certain value, the higher the quality level of that dimension under the judgment of a large language model. Secondly, calculate the confidence score of the large model dimension. The semantic confidence evaluator extracts the average output probability from the token-level probability distribution of each judgment result generated by the large language model as the basic metric for generating confidence. Judgments with an average output probability lower than a preset low confidence threshold are automatically marked as low confidence results. Subsequently, a semantic consistency check is performed on the event statement completeness judgment results. The semantic element extraction results output by the large language model are cross-validated with the keyword matching results predefined by the rule engine. If there is a significant difference between the two, the confidence of the judgment is lowered. Based on this, the overall confidence metric of the large model quality control results in each dimension is calculated.

[0013] Preferably, the collaborative decision-making module first reads the confidence weight vector corresponding to the current report type, performs short-term adaptive adjustment of the effective weights based on the confidence of the rule engine dimension and the confidence of the large model dimension, performs weighted fusion of the quality scores of the rule engine dimension and the quality scores of the large model dimension for the five dimensions, obtains the five-dimensional quality score and the overall comprehensive quality score, and outputs the collaborative decision-making result; secondly, it detects discrepancies between the quality control results of the rule engine and the quality control results of the large model, calculates the discrepancy index, adds discrepancy markers to the quality control objects with discrepancies, generates suggestions for manual review for reports exceeding a preset threshold, and updates the confidence weight vector periodically based on the historical performance after manual review, and packages the above results into a discrepancy detection result; finally, it assembles the collaborative decision-making result and the discrepancy detection result into a standardized quality control report.

[0014] Preferably, the process of obtaining collaborative decision results through weighted fusion is as follows: First, the confidence weight vector corresponding to the current report type is read from the system parameter table. This confidence weight vector contains the rule engine weight coefficients. and the weight coefficients of the large language model The confidence weight vector is adaptively adjusted to the baseline weights in the short term based on the confidence level of the current report; subsequently, the adjusted effective weights are used. and Perform step-by-step fusion calculations; secondly, for each dimension in the quality control dimension set {completeness, standardization, consistency, compliance, logic}. Perform step-by-step fusion computation: when When considering the integrity dimension, first read the core required defect marker from the rule engine's quality control result data object. If the marker is true, then the integrity score is applied. Set it directly to 0; if the flag is false, perform a weighted fusion calculation of the two results; when When the dimensions are normative, consistency, compliance, or logical, the two results are weighted and fused sequentially. and The five dimensions are weighted and fused using a confidence weight vector. Then, the fused scores from the five dimensions are directly mapped to the quality scores for each dimension. The overall comprehensive quality score is obtained by taking the arithmetic mean of the quality scores from the five dimensions. If a fatal defect is marked in the rule engine's quality control results, then If forced to zero, the report is immediately deemed unqualified; if a critical defect is marked, then... Pick Finally, the output includes score vectors for each dimension and an overall comprehensive quality score. The collaborative decision-making results.

[0015] The second aspect of this invention proposes a quality control system for adverse event reporting that integrates a rule engine and a large language model. The rule engine quality control module, the large language model quality control module, and the collaborative decision-making module of the method described in the first aspect are deployed in a quality control server. The quality control server also includes a data acquisition and preprocessing module, a quality control rule base, a system parameter table, and a quality control report generation module. The data acquisition and preprocessing module is used to acquire raw report data from multi-source systems, preprocess it into adverse event report data, parse the adverse event report data into a structured field set and an unstructured text set, and establish a cross-validation mapping relationship between the two. The quality control rule base is used to store quality control rule sets and to provide the rule data required for rule loading and matching for the rule engine quality control module. The system parameter table is used to store the confidence weight vector, low confidence threshold, high confidence threshold and divergence threshold parameters corresponding to different report types, providing the configuration data required for the collaborative decision-making module to perform fusion calculation and threshold determination; The quality control report generation module is used to assemble the collaborative decision-making results and the discrepancy detection results into a standardized quality control report.

[0016] The innovative points and beneficial effects of this invention include: (1) A dual-engine collaborative decision-making mechanism based on confidence weighted fusion is proposed. The deterministic control capability of the rule engine and the flexible semantic reasoning capability of the large language model are dynamically fused through the confidence weight vector. The rule engine is responsible for hard constraints such as integrity verification, format verification, enumeration value verification and logical consistency verification, while the large language model is responsible for flexible reasoning such as semantic accuracy verification of event description understanding and terminology normativity judgment. The two complement each other, and the fusion result retains both the auditability of the rule engine and the semantic understanding advantage of the large model. (2) Design an adaptive closed-loop optimization strategy based on the feedback of quality control results. After the quality control report is generated, the degree of divergence between the quality control results of the rule engine and the large model is used as a feedback signal to dynamically adjust the confidence weight vector. At the same time, the divergent samples are filtered and injected into the rule update queue and the large model fine-tuning dataset so that the two engines can continuously optimize their respective quality control accuracy during continuous use. (3) Establish a cross-validation mechanism between structured fields and unstructured text. By constructing a cross-validation mapping relationship between the report date, key fields of event consequences and event statement fields in the unstructured text set, the rule engine’s deterministic verification of structured fields and the semantic elements extracted from free text by the large language model can be mutually verified. In this way, cross-modal consistency constraints are introduced in addition to the independent judgment of the two engines, reducing the omissions and misjudgments caused by the limitations of a single engine perspective. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the following description is only one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the overall technical route of the present invention.

[0019] Figure 2 This is a flowchart of the multi-source adverse event reporting data collection and preprocessing process of the present invention.

[0020] Figure 3 This is a flowchart illustrating the structured field determination quality control process based on a rule engine, as described in this invention.

[0021] Figure 4 This is a flowchart of the unstructured text semantic quality control process based on a large language model, as presented in this invention.

[0022] Figure 5 This is a flowchart of the dual-engine collaborative decision-making and quality control report generation process of the present invention.

[0023] Figure 6 This is a comparison chart of the F1 scores of the three quality control schemes in various dimensions in this embodiment of the invention.

[0024] Figure 7 This is a graph showing the precision and recall rates of dual-engine collaborative quality control across various dimensions in an embodiment of the present invention.

[0025] Figure 8 This is a diagram showing the degree of divergence between the rule engine and the large model in an embodiment of the present invention. Detailed Implementation

[0026] This invention proposes a quality control method for adverse event reporting that integrates a rule engine and a large model. The overall process is as follows: Figure 1 As shown: S1, Multi-source Adverse Event Report Data Acquisition and Preprocessing: First, raw report data from medical institution reporting systems, adverse drug reaction monitoring systems, and medical device adverse event monitoring systems are received through data access interfaces. After deduplication and format unification, adverse event report data is generated. Second, based on predefined field mapping rules, the adverse event report data is parsed into a structured field set and an unstructured text set, and a cross-validation mapping relationship is established between the report date, key fields of event consequences, and event description fields. Finally, a quality control rule set is constructed based on five dimensions: completeness, standardization, consistency, compliance, and logic, and loaded into the rule base of the rule engine.

[0027] S2, Deterministic Quality Control Based on Structured Fields of the Rule Engine: Constructing a rule engine quality control module; using the structured field set and quality control rule set output from S1 as input to the rule engine quality control module; the rule engine quality control module consists of a rule loader, a rule matcher, and a confidence calculator; first, the rule loader reads the subset of quality control rules corresponding to the current report type from the quality control rule set and compiles it into an executable matching network within the rule engine; second, the rule matcher uses the rule matcher to perform deterministic matching of each field value in the structured field set with the corresponding matching network, outputting a rule engine quality control result data object, and adding a core required field defect marker when all core required fields are completely missing; finally, the confidence calculator calculates the rule engine dimension quality score and rule engine dimension confidence, packages the rule engine quality control result data object, rule engine dimension quality score, and rule engine dimension confidence into a rule engine quality control result, and passes it to the S4 collaborative decision module.

[0028] S3, Semantic Quality Control of Unstructured Text Based on a Large Language Model: This step aims to utilize a large language model to perform flexible quality control inference based on semantic understanding on an unstructured text set. The unstructured text set, quality control rule set, and cross-validation mapping relationship output from S1 are used as input to the large language model. The large language model quality control module consists of a prompt word builder, a large language model inference engine, and a semantic confidence evaluator. First, the prompt word builder transforms the quality control rule set and unstructured text set into inference prompt words for unstructured text, and simultaneously transforms the cross-validation mapping relationship into a cross-validation prompt template. Second, the large language model... The inference engine loads a pre-trained large language model to process inference prompts one by one, generating semantic quality control results for five dimensions: completeness of event statements, standardization of terminology, semantic consistency, timeliness of reports, and logical rationality. It also performs consistency comparison between structured field values ​​and unstructured text semantic elements based on cross-validation prompt templates, outputting large model quality control result data objects. Finally, the semantic confidence evaluator calculates the large model dimension quality score and the large model dimension confidence, and packages the large model quality control result data objects, the large model dimension quality score, and the large model dimension confidence into a large model quality control result, which is then passed to the S4 collaborative decision module.

[0029] S4, Dual-Engine Collaborative Decision Making and Quality Control Report Generation: A collaborative decision-making module is constructed, taking the rule engine quality control results output by S2, the large model quality control results output by S3, and a predefined confidence weight vector as input. First, the confidence weight vector corresponding to the current report type is read. Based on the confidence levels of the rule engine and large model dimensions, the effective weights are adaptively adjusted in the short term. Weighted fusion is performed on the quality scores of the rule engine and large model dimensions across the five dimensions to obtain the five-dimensional quality score and the overall comprehensive quality score, outputting the collaborative decision-making result. Second, discrepancies between the rule engine and large model quality control results are detected, and a discrepancy index is calculated. Discrepancy markers are added to quality control objects with discrepancies. For reports exceeding a preset threshold, a suggestion for manual review is generated. Based on historical performance after manual review, the confidence weight vector is updated periodically, and the above results are packaged into a discrepancy detection result. Finally, the collaborative decision-making result and the discrepancy detection result are assembled into a standardized quality control report.

[0030] The present invention will be further described below with reference to embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0031] S1. Data Acquisition and Preprocessing for Multi-Source Adverse Event Reporting To conduct comprehensive quality analysis of adverse event report data and ensure the standardization and processability of input data for subsequent quality control steps, the process establishes a standardized data processing framework through data access, field parsing, and rule construction. The specific process is as follows: Figure 2 As shown.

[0032] S1-1 Unified access to multi-source reporting data: First, the system receives electronic report data from the National Adverse Drug Reaction Monitoring System and the Medical Device Adverse Event Monitoring Information System via a REST API interface. It also supports batch import of historical report data from CSV and Excel files, which are then consolidated into an intermediate report dataset in JSON format. Second, the system performs basic field deduplication validation on each report data, using the report number as a unique identifier to remove duplicate submitted report records, and outputs the deduplicated adverse event report data. Finally, the standardized report data is persisted to the report database, and a globally unique internal processing identifier is assigned for tracking and association in subsequent quality control steps.

[0033] S1-2 Parsing Structured Fields and Unstructured Text: First, based on the pre-configured field parsing mapping table, fields with clearly defined value range constraints, such as report number, report date, report nature, product name, model / specification, registration certificate number, product batch number, production date, expiration date, usage time, implantation date, discontinuation date, event consequences, correlation evaluation, and reporter's position, are extracted from the adverse event report data and categorized and assembled into a structured field set. Second, free text fields, such as event description, preliminary cause analysis of the event, description of treatment measures, and description of the adverse event's improvement time, are extracted and categorized and assembled into an unstructured text set. Third, key fields such as report date and event consequences are established within the structured field set. The cross-validation mapping relationship between the fields and the event statement fields in the unstructured text set is as follows: the report date is used to compare the occurrence date, discovery date or awareness date extracted from the event statement text for temporal logical consistency, and the event consequences are used to compare the event consequences description extracted from the event statement text for semantic consistency. The key fields participate in the rule engine validation as fields in the structured field set, and also participate in the large model cross-validation as semantic elements that need to be extracted and compared from the unstructured text. Finally, type validation and preliminary format standardization checks are performed on each field in the structured field set to ensure that the field data type matches the predefined pattern.

[0034] S1-3 Quality Control Rule Set Construction: First, based on the regulatory requirements for adverse event monitoring of medical devices and the clinical reporting business specifications, non-empty check rules for mandatory fields are defined from the perspective of completeness. Core mandatory fields include report number, report date, event occurrence date, discovery or awareness date, product name, model / registration certificate number, event consequences, death / serious injury details, and reporting subject. Other fields are treated as ordinary mandatory fields. From the perspective of standardization, enumerated value range matching rules and date format validation rules are defined. From the perspective of consistency, cross-field logical consistency rules are defined. From the perspective of compliance, report timeliness rules are defined. From the perspective of logic, time... The initial quality control rule set is formed by first establishing the order rules and the numerical range rules. Then, each quality control rule in the initial set is structured using JSON Schema format to obtain the final quality control rule set. Rule fields include rule identifier, dimension, applicable field name, judgment logic expression, error level, and weight coefficient. The error level distinguishes between warning, general, and severe defects, and the complete absence of core required fields is separately identified by a core required field defect marker. Finally, the quality control rule set is loaded into the rule engine's rule base, and a corresponding confidence weight coefficient is configured for each rule.

[0035] S2. Structured field determination property control based on rule engine To achieve efficient deterministic validation of structured field sets and ensure that adverse event reports meet basic compliance requirements in terms of information integrity, format standardization, and field consistency, this invention designs a rule engine quality control module. This module adopts a forward chain inference architecture based on the Rete algorithm, with deterministic matching paths and auditable decision chains as design principles. The overall process is as follows: Figure 3 As shown.

[0036] S2-1 Rule compilation and matching network construction: This step takes the quality control rule set constructed in S1 as input, and the rule matcher performs a deterministic quality control judgment on each field value one by one. First, the rule loader filters a subset of rules corresponding to the current report type from the quality control rule set, based on the applicable report type identifier marked in the rule metadata. Second, the selected rule subset is parsed into rule objects containing condition and action parts. The condition part is expressed as a triplet judgment expression consisting of field name, operator, and expected value. The action part specifies the processing behavior when the rule is triggered, including error marking and score deduction. Third, based on the Rete algorithm, the rule objects are compiled into a matching network structure with a shared condition network. This matching network consists of three layers: root node, Alpha node, and Beta node. The root node is responsible for receiving fact assertions and distributing them to all downstream child nodes. The Alpha node stores the basic matching conditions of different types of nodes and performs single-condition judgment when a fact arrives. The same condition is only built once in the network to reduce computational redundancy. The Beta node performs cross-condition connection operations and achieves incremental updates by maintaining a token list of matched results. When a new fact is asserted, the system only needs to verify the new combination, thereby avoiding a full duplicate scan of the entire rule set. Finally, the compiled matching network resides in the working memory of the rule engine, waiting to be triggered for execution.

[0037] S2-2 Deterministic matching of structured fields: This step takes a structured field set and a compiled matching network as input to the rule matcher, which performs deterministic quality control judgments on each field value. First, each field value in the structured field set is sequentially injected into the working memory. The Rete algorithm propagates fact assertions of the field values ​​from top to bottom through the condition network, and each condition node of each rule is matched against the field value. Second, for integrity dimension rules, the decision expression checks whether the required field values ​​are non-empty and non-blank strings, and records the names of all missing fields. Further, for normalization dimension rules, the decision expression checks whether the enumerated value field belongs to the predefined set of valid values ​​and verifies whether the date field format matches the YYYY-MM-DD pattern. Subsequently, for consistency dimension rules... The rules are divided into several dimensions. First, for compliance-related rules, the interval between the discovery or awareness date and the reporting date is determined to be within the legally mandated reporting timeframe. Second, for logical rules, the event occurrence date is determined to be no later than the discovery or awareness date and no later than the reporting date. Finally, the rule engine summarizes the matching results of all rules and outputs a rule engine quality control result data object. This data object contains the rule identifier code and the dimension to which each rule belongs. The system considers various factors, including: a pass / fail conclusion, weighting coefficients, error levels, and intermediate calculation results such as the sum of pass / fail rule weights for each dimension and the sum of all rule weights; further, it determines whether missing fields are core required fields. If any core required field is completely missing, a core required field defect marker is added to the rule engine's quality control result data object. The system also defines a three-level defect system: Level 1 is a fatal defect, triggered by the complete absence of any core required field, and constrained by a direct report failure and an overall quality score reduction. The first level is set to zero, and the quality control report is marked with "Fatal Flaw: Report Unqualified" at the top; the second level is a critical defect, triggered by three or more missing common required fields or five or more critical defect entries, with the following constraints. The upper limit is capped at 0.79, and the quality control report is marked "Critical Defect: Recommendation to Reject and Modify"; the third level is a general defect, which is a warning or general defect that does not belong to the fatal or critical defects. It is deducted normally according to the weight and there is no overall cap.

[0038] S2-3 Rule Engine Quality Score and Confidence Calculation: This step uses the rule engine quality control result data object as input, and the confidence calculator calculates the rule engine quality score and confidence score respectively. First, the preset weight coefficient is read from the metadata of each rule. The value range of this weight coefficient is from zero to one, reflecting the relative importance of different quality control rules in the overall quality evaluation. Among them, the weight coefficient of rules with missing core required fields is the highest, and the weight coefficient of format prompt rules is the lowest. Second, the rules that pass the inspection are weighted and summed according to their weight coefficients, and the rules that fail are treated as zero. The weighted sum is then divided by the sum of the weight coefficients of all rules to obtain the dimension standardized to the zero-to-one range. The rule engine dimension quality score The closer this value is to one, the higher the pass rate of the report under the applicable rules of the rules engine; secondly, the calculation dimension Rule engine dimension confidence This confidence level is determined by the dimensions in the current report. The ratio of the number of rules that can actually be executed within a dimension to the total number of applicable rules for that dimension is determined, i.e., the dimension The rule applicability rate reflects the rule engine's applicability to dimensions. The adequacy of quality control coverage reflects the adequacy of the rule engine's quality control coverage of the report. The closer this value is to one, the more adequacy the rule engine's quality control coverage of the report and the more reliable the judgment results. Finally, the rule engine's quality control result data object and rule engine dimension quality score are combined. and the confidence level of the rule engine dimension The results are packaged into a structured rules engine quality control report and then transmitted to the S4 collaborative decision-making module for fusion processing.

[0039] S3. Semantic quality control of unstructured text based on a large language model To achieve semantic-level quality control reasoning for unstructured text sets and compensate for the limitations of rule engines in free text understanding and semantic judgment, this invention designs a large language model quality control module. This module uses a pre-trained large language model as the core of its reasoning, and transforms the quality control rule set into a semantic judgment task executable by the large language model through prompt word templates. The overall process is as follows: Figure 4 As shown.

[0040] S3-1 Semantic Quality Control Tip Construction: This step takes an unstructured text set and a quality control rule set as input to transform the quality control rules into semantic judgment instructions that the large language model can understand. First, the prompt word builder derives a semantic integrity prompt template for the integrity dimension rules, requiring the large language model to determine whether the event statement contains necessary semantic elements such as the time of adverse event occurrence, specific symptom descriptions, treatment measures taken, and final outcome. Second, for the standardization dimension rules, it derives a terminology standardization prompt template, requiring the model to identify whether the medical terms used in the event statement conform to the standard expressions in clinical diagnosis and treatment guidelines. Third, for the consistency dimension rules, it derives a semantic consistency prompt template, requiring the model to determine whether there is a reasonable causal relationship between the adverse event described in the event statement and product use. Then, for the compliance dimension rules, it derives a semantic compliance prompt template, requiring the model to determine whether the report text explicitly mentions the date of discovery or awareness of the adverse event, and accordingly determine whether the timeliness of the report complies with the statutory reporting time limit requirements. Finally, for the logic dimension rules, it derives a semantic logic prompt template, requiring the model to identify whether the time sequence involved in the event description is reasonable, and to detect whether numerical anomalies in the description of adverse event consequences violate clinical common sense. Finally, each free text segment from the unstructured text set is filled into the corresponding placeholder area of ​​the above prompt template to generate inference prompt words containing the semantic features of the text to be checked. At the same time, a cross-validation prompt template is constructed based on the cross-validation mapping relationship. Specifically, the correspondence between the report date, the key field of event consequences, and the event statement field is read from the cross-validation mapping relationship. For the report date field, a time logic consistency verification instruction is defined, requiring the model to extract the event occurrence date and the date of discovery or awareness from the event statement text and determine the chronological relationship with the report date field value in the structured field set. For the event consequences field, a semantic consistency verification instruction is defined, requiring the model to extract the event consequences description from the event statement text and determine the semantic matching with the event consequences field value in the structured field set. The structured field values ​​to be compared are filled into the structured field placeholder area in the template, and the event statement text is filled into the text extraction placeholder area in the template. The cross-validation task instructions of the above field pairs are combined into a complete cross-validation prompt template.

[0041] S3-2 Large Language Model Semantic Quality Control Inference: This step uses inference prompts as input to perform semantic quality control judgments line by line through a large language model. First, the large language model inference engine loads a pre-trained large language model based on a Transformer decoder architecture as the inference base. It then sequentially outputs each semantic judgment result through autoregressive generation, loading model weights and configuring inference hyperparameters in the localized inference environment. The temperature parameter reduces generation randomness, and the maximum output token length controls the length of the generated text. Second, the inference prompts are fed line by line into the large language model, which then sequentially outputs each semantic judgment result through autoregressive generation. Each judgment result is output as a structured JSON object containing three fields: judgment conclusion, judgment basis, and confidence score. Furthermore, regarding the completeness of the event statement, the model checks whether the text contains four semantic elements: a clear description of the event's occurrence time, a systematic description of specific clinical manifestations, a complete record of medical interventions, and a precise statement of the event's final outcome. For the standardization of terminology, the model implicitly matches the medical terms in the event description with a standard medical terminology database, marking terms with non-standard expressions or colloquial usage. For semantic consistency, the model analyzes whether the causal relationship between the adverse event and product use in the event description is consistent, checks the logical match between the event consequence description and the clinical diagnosis, and marks entries with ambiguous or contradictory causal relationships. For the timeliness of the report, the model extracts time elements such as the event occurrence date and the date of discovery or awareness from the event statement, determines whether the report submission date is within the statutory reporting deadline, and marks the specific number of days of overdue reporting. For logical reasonableness, the model checks whether the time sequence in the event description is reasonable, and also detects whether the numerical descriptions violate clinical common sense or exceed normal physiological ranges. Finally, based on the cross-validation prompt template constructed in S3-1, the consistency between structured fields and unstructured text is determined. The model extracts key semantic information such as the report date, the date of discovery or awareness, and the consequences of the event from the event statement text, and compares them item by item with the corresponding field values ​​in the structured field set, marking the judgment items where the field values ​​contradict the text description; then, all semantic judgment results and cross-validation results are collected, and a large model quality control result data object of the unstructured text set is output.

[0042] S3-3 Large Model Quality Score and Confidence Assessment: This step uses the large model quality control results data object as input to calculate the quality score and confidence level of the large language model in each dimension. The two are clearly distinguished: the quality score reflects the large language model's evaluation of the report quality, and the confidence level reflects the reliability of that evaluation. First, the quality scores for each dimension of the large model are calculated. For large language models in terms of dimensionality For each semantic quality control judgment result, extract its judgment conclusion (pass or fail) and corresponding defect level. Adopt the same quality scoring logic as the rule engine: for items that pass the judgment, perform a weighted sum based on preset item weight coefficients; for items that fail the judgment, treat them as zero values, and divide the weighted sum by the dimension. The sum of the weights of all decision items is used to obtain a large model dimensionality quality score standardized to the zero-to-one interval. The closer this value is to a certain level, the higher the quality level of that dimension in the judgment of a large language model; dimension The sum of the weights of the items judged by the internal criteria reflects the contribution of the quality control points deemed qualified by the large language model to the overall quality evaluation. Secondly, the confidence scores of the large model dimensions are calculated. The semantic confidence evaluator extracts the average output probability from the token-level probability distribution of each judgment result generated by the large language model as the basic metric for generating confidence. Judgments with an average output probability lower than a preset low confidence threshold are automatically marked as low-confidence results. Subsequently, a semantic consistency check is performed on the event statement completeness judgment results. The semantic element extraction results output by the large language model are cross-validated with the keyword matching results predefined by the rule engine. If there is a significant difference between the two, the confidence of that judgment is lowered. Based on this, the overall confidence metric of the large model quality control results across various dimensions is calculated, and the calculation formula is expressed as: ; in For large language models in dimensionality The confidence metric on the scale ranges from 0 to 1. The closer the value is to 1, the more reliable the judgment result of the large language model on this dimension is. For dimension The mean of the output probabilities of each decision within the range; α is the semantic consistency score for dimension d, which is determined by the degree of consistency between the semantic elements output by the large language model and the keyword matching results of the rule engine; α is the balance coefficient, with a default value of 0.7. It is only used to measure the reliability of the judgment results of large language models and is not directly used as a quality score in the final fusion calculation. The quality score is determined by... Independently assume responsibility. Then, mark the judgment results below the confidence threshold with a low confidence level for manual review, and remove the judgment items with confidence levels below the threshold from... In the calculation, data objects are removed or weighted down to avoid low-reliability judgments affecting the accuracy of the overall quality score. Finally, the large model quality control results data objects and large model dimensional quality scores are combined. and confidence of large model dimensions The results are packaged into a structured large model quality control report and then transmitted to the S4 collaborative decision-making module.

[0043] S4, Dual-engine Collaborative Decision Making and Quality Control Report Generation To achieve deep integration of the quality control results from the rule engine and the large model, resulting in a more comprehensive and reliable final judgment than a single quality control engine, this invention designs a collaborative decision-making module. The overall data processing process is as follows: Figure 5 As shown.

[0044] S4-1 Confidence-weighted fusion calculation: This step takes the rule engine quality control results output by S2, the large model quality control results output by S3, and the confidence weight vectors corresponding to each report type predefined in the system parameter table as inputs to perform five-dimensional collaborative decision fusion calculation. First, it reads the confidence weight vector corresponding to the current report type from the system parameter table. This confidence weight vector contains the rule engine weight coefficients. and the weight coefficients of the large language model The confidence weight vector performs short-term adaptive adjustment of the baseline weights based on the confidence level of the current report. Specifically, the confidence levels of each dimension of the rule engine are read from the rule engine quality control results output by S2. Read the confidence scores of each dimension of the large model from the large model quality control results output by S3. For each dimension The effective weights are calculated according to the following rules: If and All are greater than or equal to the preset high confidence threshold. , The default value is 0.7, so the baseline weight is used directly. and Integrate without adjustment; if Below and Higher than or equal to This indicates that the rule engine's coverage of this dimension is insufficient, while the large language model's judgment is reliable. Therefore, the effective weight of the rule engine should be reduced to [value missing]. The effective weights of the large language model were adjusted accordingly. ;like Below and Higher than or equal to This indicates that the large language model's judgment reliability is insufficient while the rule engine's coverage is sufficient. Therefore, the effective weights of the large language model should be symmetrically reduced to... The effective weight of the rules engine will be adjusted accordingly. ;like and All below If the confidence level is low, it indicates that both engines lack confidence in this dimension, marking it as having low confidence and highlighting it in the quality control report, suggesting manual review. During fusion, the baseline weights will be applied. The adjusted effective weights are constrained to the range of 0.1 to 0.9 to prevent any single engine from being completely excluded. Subsequently, the adjusted effective weights are used. and Perform step-by-step fusion calculations; secondly, for each dimension in the quality control dimension set {completeness, standardization, consistency, compliance, logic}. Perform step-by-step fusion computation: when When considering the integrity dimension, first read the core required defect marker from the rule engine's quality control result data object. If the marker is true, then the integrity score is applied. Set it directly to 0; if the flag is false, perform a weighted fusion calculation of the two results; when When the dimensions are normative, consistency, compliance, or logical, the two results are weighted and fused sequentially. and The fusion formula, which uses a confidence weight vector for weighted fusion, is as follows: ; in For dimension The fusion score is then used to determine the overall quality score across the five dimensions. Based on this, the fusion score across the five dimensions is directly mapped to the quality score for each dimension, specifically, the completeness score. equal Normative scoring equal Consistency score equal Compliance score equal Logical scoring equal ,each This constructs a five-dimensional quality score vector. Then, the arithmetic mean of the quality scores across the five dimensions is taken to obtain the overall comprehensive quality score. Its calculation formula is expressed as ; in For the overall quality score, The arithmetic mean of the scores across the five dimensions, with the denominator being 5 (the total number of dimensions), reflects the overall quality level of the report across all quality control dimensions. If a critical flaw is flagged in the rule engine's quality control results, then... If forced to zero, the report is immediately deemed unqualified. If a critical defect is marked, then... Pick Finally, the output includes score vectors for each dimension and an overall comprehensive quality score. The collaborative decision-making results.

[0045] S4-2 Disagreement Detection and Manual Verification Marking: This step takes the quality control results from the rule engine, the large model, and the collaborative decision-making as inputs to detect discrepancies between the two engine quality control results and provide auxiliary information for manual review. First, using each field in the structured field set as the quality control object, it iterates through the rule engine's judgment on the structured fields and the large model's judgment on the consistency between structured fields and unstructured text. If the two judgments are inconsistent (one marked as passed while the other is marked as failed), a discrepancy flag is added to the quality control object. Second, the discrepancy index is calculated. This indicator is the proportion of the number of disputed judgment items to the total number of judgment items, and its calculation relationship is expressed as follows: ; Where is the divergence index, represents the number of divergent judgment items, and represents the total number of judgment items; the value range is from zero to one. Next, regarding the divergence index... Exceeding the preset divergence threshold The report prominently displays a suggestion for manual review in the quality control report, and lists the dual-engine judgment criteria for discrepancies to facilitate comparison and judgment by reviewers. Subsequently, the confidence weight vector is automatically updated periodically, based on the historical performance of the dual engines after manual review and confirmation. Specifically, in each update cycle... Update cycle The default is to process a cumulative total of 500 reports or once per calendar month, whichever comes first. Statistical dimensions. The historical accuracy rates of the rule engine and the large language model were calculated based on the number of correctly judged entries verified by manual review. and Update the baseline weights as follows: ; ; in For the adjusted dimensions The weighting coefficients of the rule engine, For the adjusted dimensions The weight coefficients of the language model at Shanghai University The smoothing coefficient is set to 0.8 by default. (After update) Cropped to [0.1, 0.9]. The weights are simultaneously reduced to [0.1, 0.9] to prevent a single engine's weight from degrading to zero. If the overall quality control accuracy drops by more than two percentage points for two consecutive cycles after the update, it will automatically roll back to the previous stable version.

[0046] Finally, the updated confidence weight vector is written into the system parameter table, and the data of the divergence entries is fed back to the system maintenance interface. After review and confirmation, it is used for subsequent updates of the quality control rule set or expansion of the large language model fine-tuning dataset.

[0047] S4-3 Five-Dimensional Quality Control Report Generation: This step uses the collaborative decision-making results and discrepancy detection results as input to assemble a standardized quality control report with a specified format. First, based on the completeness dimension, a comprehensive completeness score is extracted from the collaborative decision-making results. This score is then used to summarize the non-empty check results and semantic completeness judgment results for all required fields under this dimension, listing the names and types of all missing fields. Furthermore, the core required field defect markers retained from the collaborative decision-making results are prominently displayed in the completeness defect list to indicate the presence of completely missing core required fields in the report. Second, based on the normativity dimension, a comprehensive normativity score is extracted from the collaborative decision-making results. This score is then used to summarize the enumerated value matching results and terminology normativity judgment results under this dimension, listing the names and expected formats of all improperly filled fields to form a normativity defect list. Third, based on the consistency dimension, a comprehensive consistency score is extracted from the collaborative decision-making results. This score is then used to summarize the cross-field logical consistency judgment results and semantic consistency judgment results under this dimension, listing all inconsistent field pairs and conflicting descriptions to form a consistency defect list. Then, based on the compliance dimension, a comprehensive compliance score is extracted from the collaborative decision-making results as the compliance score. The results of report timeliness assessment and the matching results between report nature and report deadline are summarized under this dimension, and a list of all overdue reports by the specific number of days is generated to form a compliance defect list. Next, based on the logic dimension, a comprehensive logic score is extracted from the collaborative decision-making results as the logic score. The results of time sequence verification and numerical range verification under this dimension are summarized, and a list of all field data that violates business logic is generated to form a logic defect list. Finally, the quality scores of the above five dimensions, the overall comprehensive quality score, the defect item lists for each dimension, and the discrepancy detection tags are assembled into a standardized quality control report containing a cover summary, detail pages for each dimension, and a defect summary. This report is output simultaneously in both PDF and JSON formats for use by downstream monitoring and analysis systems.

[0048] S5, Model Deployment The rule engine quality control module, the large language model quality control module, and the collaborative decision-making module are deployed on the quality control server. The quality control server also includes a data acquisition and preprocessing module, a quality control rule base, a system parameter table, a quality control report generation module, and a result output interface. At the same time, the data acquisition and preprocessing module is connected to the medical institution reporting system, the adverse drug reaction monitoring system, and the medical device adverse event monitoring system through a standard data interface. The data acquisition and preprocessing module is used to acquire original report data from multiple sources in real time or at regular intervals through REST API interface or batch file import method. After performing deduplication and format unification on the original report data, it forms adverse event report data. Based on predefined field mapping rules, it parses the adverse event report data into a structured field set and an unstructured text set, and establishes a cross-validation mapping relationship between the structured field set and the unstructured text set. The quality control rule base is used to store a set of quality control rules expressed in JSON Schema format, covering five dimensions: completeness, standardization, consistency, compliance, and logic, and to provide the rule data required for rule loading and matching for the rule engine quality control module; The system parameter table is used to store the confidence weight vector, low confidence threshold, high confidence threshold and divergence threshold parameters corresponding to different report types, providing the configuration data required for the collaborative decision-making module to perform fusion calculation and threshold determination; The structured field set and quality control rule set are input into the rule engine quality control module to obtain the rule engine quality control results. The unstructured text set, quality control rule set and cross-validation mapping relationship are input into the large language model quality control module to obtain the large model quality control results. The rule engine quality control results, the large model quality control results and the confidence weight vector in the system parameter table are input into the collaborative decision-making module to obtain the collaborative decision-making results and the divergence detection results. The quality control report generation module is used to assemble the collaborative decision-making results and the discrepancy detection results into a standardized quality control report that includes a cover summary, detail pages for each dimension, and a defect summary. The report is then output to the downstream monitoring and analysis system in PDF and JSON formats through the result output interface.

[0049] Experimental verification and analysis: To verify the effectiveness of the proposed method for quality control of adverse event reporting through the collaboration of a rule engine and a large model, this experiment used 5,000 adverse event reports of intrauterine devices provided by a provincial medical device adverse event monitoring center as experimental data. The experimental environment was based on the Linux operating system, and the Qwen2.5 7B parameter version large language model was loaded using the PyTorch deep learning framework. This model is based on the Transformer decoder architecture and adopts the SwiGLU activation function, Rotary Position Embedding (RoPE), and Grouped Query Attention (GQA) mechanism. The hidden layer dimension is 4096, the number of attention heads is 32, and the number of Transformer layers is 28. The training loss function is the autoregressive cross-entropy loss function, and the optimizer is AdamW with an initial learning rate set to 1e-5. Inference was performed on an NVIDIA A100 80GB GPU, the temperature parameter was set to 0.1, and the maximum output token length was set to 512. The rule engine was implemented using Python rule-engine version 4.5.3. The experiment compares and evaluates the quality control performance of three schemes—rule engine-based quality control, large language model-based quality control, and dual-engine collaborative quality control of this invention—from three dimensions: quality control accuracy, recall, and overall F1 score.

[0050] analyze Figure 6 As can be seen, the dual-engine collaborative quality control scheme proposed in this invention significantly outperforms the rule engine-only quality control and the large language model-only quality control schemes in all five quality control dimensions in terms of F1 scores. The improvement is most significant in the consistency dimension, where the collaborative scheme achieves an F1 score of 0.91, an absolute improvement of 11.0 percentage points compared to the rule engine and 5.0 percentage points compared to the large language model. In the compliance dimension, the collaborative scheme achieves an F1 score of 0.95, fully preserving the deterministic verification advantage of the rule engine in this dimension. The average F1 score across all five dimensions reaches 0.92, a substantial improvement over the rule engine's 0.88 and the large model's 0.76, validating the effectiveness of the dual-engine collaborative mechanism.

[0051] analyze Figure 7It can be seen that the dual-engine collaborative solution maintains a good balance between precision and recall across all five quality control dimensions. In the prescriptive dimension, precision reaches 0.95 and recall reaches 0.93, with the smallest difference between the two, indicating that the rule engine has stable deterministic judgment capabilities in format validation and enumeration value matching. In the logical dimension, precision is 0.88 and recall is 0.86. Since some logical rules involve time extrapolation and numerical range determination, the collaborative solution, by introducing semantic validation assistance from a large language model, improves the recall rate by 3.0 percentage points compared to the pure rule engine. Overall, precision and recall are all above 0.86 across all dimensions, demonstrating that the collaborative solution effectively improves recall performance while maintaining high precision.

[0052] analyze Figure 8 It can be seen that the rule engine and the large language model exhibit differentiated discrepancies across different quality control dimensions. The consistency dimension shows the highest discrepancy rate at 14.7, indicating a significant complementary space between deterministic rules and semantic reasoning in cross-field semantic consistency judgment. This is a key scenario where the dual-engine collaborative mechanism demonstrates its value. The logic dimension shows a discrepancy rate of 11.3, reflecting ambiguity in some logical judgment rules at the textual description level. The completeness and normativity dimensions show lower discrepancy rates of 8.2 and 6.5 respectively, indicating that the judgments of the two types of engines tend to be consistent in these dimensions. The compliance dimension shows the lowest discrepancy rate at only 5.1, because compliance rules are based on clear regulatory deadlines and enumerated values, and the deterministic judgments of the rule engine are highly consistent with the semantic understanding conclusions of the large model. Overall, the differences in the discrepancy distribution confirm that the rule engine and the large language model have significant complementary characteristics in quality control capabilities.

[0053] Table 1. Comparison of accuracy and recall rates of the three quality control schemes across various dimensions.

[0054] Analysis of the experimental results table shows that the dual-engine collaborative quality control method proposed in this invention outperforms the single-engine quality control schemes of rule engines or large language models in terms of accuracy and recall across all five quality control dimensions. In the completeness dimension, the collaborative scheme achieves an accuracy of 0.93, a 4.0 percentage point improvement over the rule engine and a 15.0 percentage point improvement over the large model. In the normativity and compliance dimensions, the rule engine has a natural advantage; the collaborative scheme, by retaining the dominant role of the rule engine while incorporating the auxiliary judgment of the large model, achieves accuracies of 0.95 and 0.96, respectively. In the consistency dimension, the semantic understanding capability of the large language model enables a recall of 0.88; the collaborative scheme further improves the recall to 0.90. In the logic dimension, the collaborative scheme achieves an accuracy of 0.88 and a recall of 0.86. By introducing the semantic verification assistance of the large language model, the recall is improved by 3.0 percentage points compared to the pure rule engine, verifying the complementary effect of the collaborative mechanism in temporal logic and numerical range determination. The combined F1 score improved from 0.88 for the rule engine and 0.76 for the large model to 0.92 for the collaborative solution, verifying that the dual-engine collaborative mechanism can effectively integrate the complementary advantages of the two types of quality control capabilities and achieve an overall improvement in quality control performance.

[0055] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0056] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for quality control of adverse event reporting that integrates a rule engine and a large model, characterized in that, Includes the following processes: Acquire raw report data from multi-source systems and preprocess it into adverse event report data; Adverse event report data is parsed into a set of structured fields and a set of unstructured text, and a cross-validation mapping relationship is established between the two. At the same time, a set of quality control rules is constructed. Input the structured field set and quality control rule set into the rule engine quality control module, and output the rule engine quality control results, including the rule engine quality control result data object, the rule engine dimension quality score and the rule engine dimension confidence. The unstructured text set, quality control rule set, and cross-validation mapping relationship are input into the large language model quality control module. Prompt word generation, prompt word conversion, and confidence evaluation are performed respectively to obtain the large model quality control results, which include large model quality control result data objects, large model dimension quality scores, and large model dimension confidence. The rule engine quality control results, large model quality control results, and predefined confidence weight vectors are input into the collaborative decision-making module. The confidence weight vector corresponding to the current report type is read. The collaborative decision-making results are obtained by performing a weighted fusion of the rule engine dimension quality score and the large model dimension quality score based on the confidence of the rule engine dimension and the confidence of the large model dimension. Discrepancies between the rule engine quality control results and the large model quality control results are detected, and the discrepancy detection results are calculated.

2. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 1, characterized in that: The process of parsing adverse event report data into a structured field set and an unstructured text set specifically involves: First, based on the pre-configured field parsing mapping table, fields with clearly defined value range constraints, including report number, report date, report nature, product name, model and specifications, registration certificate number, product batch number, production date, expiration date, usage time, implantation date, discontinuation date, event consequences, correlation evaluation, and reporter's position, are extracted from the adverse event report data and categorized and assembled into a structured field set. Second, free text fields, including event description, preliminary cause analysis of the event, description of treatment measures, and description of the time of improvement of the adverse event, are extracted from this data and categorized and assembled into an unstructured text set.

3. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 1, characterized in that: The construction of the quality control rule set specifically involves: First, based on the regulatory requirements for adverse event monitoring of medical devices and the clinical reporting business specifications, non-empty check rules for mandatory fields are defined from the perspective of completeness. Core mandatory fields include report number, report date, event occurrence date, discovery or awareness date, product name, model / registration certificate number, event consequences, death / serious injury details, and reporting subject. Other fields are treated as ordinary mandatory fields. From the perspective of standardization, enumerated value range matching rules and date format validation rules are defined. From the perspective of consistency, cross-field logical consistency rules are defined. From the perspective of compliance, report timeliness rules are defined. From the perspective of logic, time... The initial quality control rule set is formed by first establishing the order rules and the numerical range rules. Then, each quality control rule in the initial set is structured using JSON Schema format to obtain the final quality control rule set. Rule fields include rule identifier, dimension, applicable field name, judgment logic expression, error level, and weight coefficient. The error level distinguishes between warning, general, and severe defects, and the complete absence of core required fields is separately identified by a core required field defect marker. Finally, the quality control rule set is loaded into the rule engine's rule base, and a corresponding confidence weight coefficient is configured for each rule.

4. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 1, characterized in that: The rule engine quality control module consists of a rule loader, a rule matcher, and a confidence calculator. First, the rule loader reads a subset of quality control rules corresponding to the current report type from the quality control rule set and compiles it into an executable matching network within the rule engine. Second, the rule matcher performs deterministic matching between each field value in the structured field set and the corresponding matching network, outputting a rule engine quality control result data object, and adding a core required field defect marker when all core required fields are completely missing. Finally, the confidence calculator calculates the rule engine dimension quality score and the rule engine dimension confidence score, and packages the rule engine quality control result data object, the rule engine dimension quality score, and the rule engine dimension confidence score into a rule engine quality control result.

5. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 4, characterized in that: The calculation of the rule engine dimension quality score and rule engine dimension confidence using a confidence calculator is as follows: First, a preset weight coefficient is read from the metadata of each rule. This weight coefficient ranges from zero to one, reflecting the relative importance of different quality control rules in the overall quality evaluation. Rules with missing core required fields have the highest weight coefficient, while rules with formatting prompts have the lowest. Second, rules that pass the check are summed according to their weight coefficients, while rules that fail are treated as zero. The weighted sum is then divided by the sum of all rule weight coefficients to obtain the dimension standardized to the zero-to-one range. The rule engine dimension quality score The closer this value is to one, the higher the pass rate of the report under the applicable rules of the rules engine; secondly, the calculation dimension Rule engine dimension confidence This confidence level is determined by the dimensions in the current report. The ratio of the number of rules that can actually be executed within a dimension to the total number of applicable rules for that dimension is determined, i.e., the dimension The rule applicability rate reflects the rule engine's applicability to dimensions. The adequacy of quality control coverage reflects the extent to which the rule engine covers the quality control of the report. The closer the value is to one, the more adequacy the quality control coverage of the report by the rule engine.

6. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 1, characterized in that: The large language model quality control module consists of a prompt word builder, a large language model inference engine, and a semantic confidence evaluator. First, the prompt word builder transforms the quality control rule set and unstructured text set into inference prompt words for unstructured text, while simultaneously converting cross-validation mapping relationships into cross-validation prompt templates. Second, the large language model inference engine loads a pre-trained large language model to process each inference prompt word, generating semantic quality control judgment results across five dimensions: event statement completeness, terminology standardization, semantic consistency, report timeliness, and logical rationality. Based on the cross-validation prompt templates, it performs a consistency comparison between structured field values ​​and unstructured text semantic elements, outputting the large model quality control result data object. Finally, the semantic confidence evaluator calculates the large model dimensional quality score and the large model dimensional confidence, packaging the large model quality control result data object, the large model dimensional quality score, and the large model dimensional confidence into the large model quality control result.

7. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 6, characterized in that: The calculation of the large model dimension quality score and large model dimension confidence using a semantic confidence evaluator is specifically as follows: First, calculate the quality score for the large model dimensions. For large language models in terms of dimensionality For each semantic quality control judgment result, its judgment conclusion and corresponding defect level are extracted. A quality scoring logic consistent with the rule engine is adopted: entries that pass the judgment are weighted and summed according to preset entry weight coefficients; entries that fail the judgment are treated as zero values, and the weighted sum is divided by the dimension. The sum of the weights of all decision items is used to obtain a large model dimensionality quality score standardized to the zero-to-one interval. The closer the value is to a certain value, the higher the quality level of that dimension under the judgment of a large language model. Secondly, calculate the confidence score of the large model dimension. The semantic confidence evaluator extracts the average output probability from the token-level probability distribution of each judgment result generated by the large language model as the basic metric for generating confidence. Judgments with an average output probability lower than a preset low confidence threshold are automatically marked as low confidence results. Subsequently, a semantic consistency check is performed on the event statement completeness judgment results. The semantic element extraction results output by the large language model are cross-validated with the keyword matching results predefined by the rule engine. If there is a significant difference between the two, the confidence of the judgment is lowered. Based on this, the overall confidence metric of the large model quality control results in each dimension is calculated.

8. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 1, characterized in that: The collaborative decision-making module first reads the confidence weight vector corresponding to the current report type, performs short-term adaptive adjustment of the effective weights based on the confidence of the rule engine dimension and the large model dimension, and performs weighted fusion of the quality scores of the rule engine dimension and the large model dimension of the five dimensions to obtain the five-dimensional quality score and the overall comprehensive quality score, and outputs the collaborative decision-making result. Second, it detects discrepancies between the quality control results of the rule engine and the quality control results of the large model, calculates the discrepancy index, adds discrepancy markers to the quality control objects with discrepancies, generates suggestions for manual review for reports exceeding a preset threshold, and updates the confidence weight vector periodically based on the historical performance after manual review, and packages the above results into a discrepancy detection result. Finally, it assembles the collaborative decision-making result and the discrepancy detection result into a standardized quality control report.

9. The method for quality control of adverse event reporting that combines rule engine and large model as described in claim 8, characterized in that: The process of obtaining collaborative decision-making results through weighted fusion is as follows: First, the confidence weight vector corresponding to the current report type is read from the system parameter table. This confidence weight vector contains the rule engine weight coefficients. and the weight coefficients of the large language model The confidence weight vector is adaptively adjusted to the baseline weights in the short term based on the confidence level of the current report. Then, the adjusted effective weights are used. and Perform step-by-step fusion calculations; secondly, for each dimension in the quality control dimension set {completeness, standardization, consistency, compliance, logic}. Perform step-by-step fusion computation: when When considering the integrity dimension, first read the core required defect marker from the rule engine's quality control result data object. If the marker is true, then the integrity score is applied. Set it directly to 0; if the flag is false, perform a weighted fusion calculation of the two results; when When the dimensions are normative, consistency, compliance, or logical, the two results are weighted and fused sequentially. and The five dimensions are weighted and fused using a confidence weight vector. Then, the fused scores from the five dimensions are directly mapped to the quality scores for each dimension. The overall comprehensive quality score is obtained by taking the arithmetic mean of the quality scores from the five dimensions. If a fatal defect is marked in the rule engine's quality control results, then If forced to zero, the report is immediately deemed unqualified; if a critical defect is marked, then... Pick Finally, the output includes score vectors for each dimension and an overall comprehensive quality score. The collaborative decision-making results.

10. A quality control system for reporting adverse events that integrates a rule engine and a large model, characterized in that: The rule engine quality control module, the large language model quality control module, and the collaborative decision-making module of any one of claims 1 to 9 are deployed in the quality control server. The quality control server also includes a data acquisition and preprocessing module, a quality control rule base, a system parameter table, and a quality control report generation module. The data acquisition and preprocessing module is used to acquire raw report data from multi-source systems, preprocess it into adverse event report data, parse the adverse event report data into a structured field set and an unstructured text set, and establish a cross-validation mapping relationship between the two. The quality control rule base is used to store the quality control rule set and to provide the rule data required for rule loading and matching for the rule engine quality control module. The system parameter table is used to store the confidence weight vector, low confidence threshold, high confidence threshold and divergence threshold parameters corresponding to different report types, providing the configuration data required for the collaborative decision-making module to perform fusion calculation and threshold determination; The quality control report generation module is used to assemble the collaborative decision-making results and the discrepancy detection results into a standardized quality control report.

Images