A scientific research data quality checking method and system based on AI machine learning

Abstract

The application discloses a scientific research data quality checking method and system based on AI machine learning, and relates to the technical field of data quality checking.The scientific research data quality checking method based on AI machine learning collects heterogeneous scientific research data, constructs a hierarchical structure data through semantic knowledge base construction and field automatic mounting, mines constraint edges and logical edges to form a dependency relationship graph, adopts a hierarchical heterogeneous identification model including a rule checker, a longitudinal trend predictor, a domain alignment detector and a dependency graph detector to perform abnormality identification, generates rule violation residuals, time series prediction residuals, cross-center alignment residuals and topological reconstruction residuals, and performs comprehensive processing to generate a comprehensive quality score and automatically divide problem levels.The application encrypts and stores evidence chains for problems meeting preset levels, and generates an inspection report, so that hidden data problems can be accurately dug out, the situation of missed judgment and misjudgment is reduced, and data checking is more comprehensive and meticulous.

Description

Technical Field

[0001] This invention relates to the field of data quality verification technology, specifically to a method and system for verifying the quality of scientific research data based on AI machine learning. Background Technology

[0002] In clinical research, basic medical experiments, and multi-center research projects, data quality is a core element in ensuring the authenticity and reliability of research conclusions. Research data is usually stored in the form of structured databases, covering multiple dimensions such as patient demographic information, visit records, laboratory tests, efficacy evaluation, and adverse events. The data is large in scale, comes from diverse sources, and is updated frequently.

[0003] In terms of data quality verification, traditional methods mainly rely on manual sampling, source data verification, and automatic verification based on fixed rules. Rule verification usually includes null value checks, format verification, numerical range constraints, and visit window verification, which can effectively identify obvious input errors or logical contradictions. In recent years, some systems have begun to introduce simple statistical analysis or outlier detection methods to help discover abnormal distributions or extreme values. In addition, for multi-center studies, some platforms provide data summarization and descriptive statistical comparison functions to help data administrators discover differences between centers.

[0004] The limitations of existing technologies include at least the following problems: Under current technologies, scientific research data quality checks generally rely on manual configuration of fixed verification rules, which can only screen for explicit errors such as null values, formatting, and out-of-bounds values ​​in single fields and single records. It is difficult to rely on the inherent hierarchical relationships between patients, visits, forms, and fields in scientific research data to uncover hidden logical anomalies across fields, forms, visits, and even multiple centers. As a result, it is difficult to learn the normal correlation and change patterns of indicators under different research centers and different visit time series, and it is also difficult to distinguish between reasonable differences in data entry habits and real data distribution deviations between multiple centers. It is easy to miss hidden logical defects and misjudge differences in normal center data. Furthermore, there is a lack of quantitative measurement methods for the degree of deviation of various types of anomalies. It can only output anomaly items in general and cannot accurately distinguish the severity and location of problems, which is difficult to meet the accuracy requirements of data quality control for multi-center scientific research projects. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a scientific research data quality verification method and system based on AI machine learning, which solves the problem that existing technologies struggle to identify cross-dimensional hidden anomalies and roughly locate problem records.

[0006] To achieve the above objectives, this invention provides the following technical solution: a scientific research data quality verification method based on AI machine learning, comprising the following steps: collecting heterogeneous scientific research data, constructing a hierarchical data structure through semantic knowledge base construction and automatic field mounting, including a patient layer, visit layer, form layer, and field layer; performing dependency mining on the hierarchical data to construct a dependency graph containing an edge set, the edge set including dependency constraint edges and logical edges; and using a pre-constructed hierarchical heterogeneity identification model to identify anomalies based on the dependency graph and the hierarchical data, including:

[0007] The system comprises the following components: a rule validator, a rule violator, and a domain alignment detector. The rule validator traverses the constraint edges in the dependency graph and performs violation verification, generating rule violation residuals. A longitudinal trend predictor, based on the dependency graph and hierarchical data, statistically analyzes the longitudinal sequence of visits and performs cross-visit temporal attention encoding and trend extrapolation to generate temporal prediction residuals. A domain alignment detector extracts a distributionally consistent field set from the dependency graph and hierarchical data, calculates reconstruction errors and dynamically adaptive threshold corrections using adversarial training to learn center-invariant features, and generates cross-center alignment residuals. A dependency graph detector constructs a heterogeneous graph based on hierarchical data and the dependency graph, performs graph structure propagation and iterative reconstruction, and generates topology reconstruction residuals. All residuals are comprehensively processed to generate a comprehensive quality score and automatically classify problem levels. Problems meeting preset levels are encrypted and stored as evidence chains, and an audit report is generated.

[0008] Furthermore, the specific steps for constructing hierarchical data are as follows: Heterogeneous scientific research data are extracted to a temporary storage area; each data table is retrieved from the temporary storage area and categorized according to a preset hierarchical mapping rule; the categorized data tables are organized into an initial structure, and a semantic knowledge base containing standard concepts, field alias mappings, unit conversion rules, and default quality control rules is pre-constructed; the initial structure is vectorized and semantically matched with the standard concepts in the semantic knowledge base through semantic similarity calculation; based on the semantic matching results, semantic mounting and rule inheritance are performed and backfilled into the initial structure to construct hierarchical data.

[0009] Further, the specific steps for constructing the dependency graph are as follows: Extract several fields from the field layer and use them as field nodes; add dependency constraint edges between field nodes that meet preset logical constraints, including time order dependencies, field logical dependencies, and inter-form consistency dependencies; perform data cleaning and quality pre-screening on the hierarchical data to generate qualified research data; based on the qualified research data, use statistical dependency mining and causal structure learning methods to generate logical edges for different fields; assign distribution consistency labels to field nodes that meet preset cross-center consistency conditions; combine the constraint edges, logical edges, and each field node and its corresponding distribution consistency label to form the dependency graph.

[0010] Furthermore, the specific steps for generating rule violation residuals are as follows: determine the constraint type and corresponding constraint threshold of the constraint edge; and based on the dependency graph, extract the set of associated field nodes that have constraint edges with the field node; determine whether the field node and its corresponding associated field node set satisfy the constraint type and corresponding constraint threshold of the constraint edge; if not, perform constraint deviation mapping processing on the field node, its corresponding associated field node set, and the constraint threshold to generate rule violation residuals.

[0011] Further, the specific steps for generating time-series prediction residuals are as follows: Based on the field nodes in the dependency graph and the visit layer and field layer in the hierarchical data, the target field is determined; for the target field, the visit longitudinal sequence is extracted in the order of visit time; the visit longitudinal sequence is input into the time-series prediction model, which uses cross-visit time-series attention encoding to perform weighted aggregation on the visit longitudinal sequence to obtain the field prediction value and the field prediction confidence interval; from the hierarchical data, the field value of the target field is extracted, and confidence residual mapping is performed with the field prediction value and the field prediction confidence interval to generate the time-series prediction residual.

[0012] Further, the specific steps for generating cross-center alignment residuals are as follows: Based on the field nodes labeled as having consistent distribution, extract the consistent field set from the hierarchical data and group them; for the grouping results, extract historical research data from the hierarchical data and generate a dynamic adaptive threshold through kernel density estimation; construct an adversarial domain adaptation network containing a feature extractor, a center discriminator, and a reconstructor, learn irrelevant feature representations through adversarial training, and calculate the reconstruction error to obtain the domain alignment residuals; jointly correct the dynamic threshold residuals and the domain alignment residuals, and use the joint correction result as the cross-center alignment residuals.

[0013] Furthermore, the construction steps of the adversarial domain adaptation network are as follows: The grouping results are input into the feature extractor, and the latent space feature vector is output after deep feature encoding; the latent space feature vector is simultaneously input into the center discriminator and the reconstructor; the center discriminator predicts the center label to which the latent space feature vector belongs and calculates the classification loss; the reconstructor reconstructs the latent space feature vector, outputs the reconstructed feature vector, and calculates the reconstruction loss; a gradient reversal layer is inserted between the center discriminator and the feature extractor, and the gradient of the classification loss is backpropagated to the feature extractor through the gradient reversal layer, so that the feature extractor learns irrelevant feature representations; the feature extractor, center discriminator and reconstructor are adversarially trained with the joint optimization objective of minimizing the reconstruction loss and maximizing the classification loss.

[0014] Furthermore, the specific steps for generating the topology reconstruction residual are as follows: Based on the hierarchical data and combined with the constraint edges between field nodes in the dependency graph, a heterogeneous graph is generated, where the connection relationship of the heterogeneous graph is determined by the constraint edges; along the connection relationship of the graph structure, multiple rounds of feature propagation and aggregation are performed through a pre-trained inductive graph neural network to generate hidden layer features; based on the hidden layer features, Monte Carlo random deactivation is enabled in the inductive graph neural network for multiple forward propagations to obtain multiple reconstruction values, and bias normalization processing is performed to generate the topology reconstruction residual.

[0015] Furthermore, the specific steps for generating the comprehensive quality score are as follows: adaptive weighting is applied to the rule violation residuals, time series prediction residuals, cross-center alignment residuals, and topology reconstruction residuals to generate a basic quality risk score; and the importance weights of fields and the integrity coefficient of the evidence chain are extracted from the field layer; based on the importance weights of fields and the integrity coefficient of the evidence chain, the basic quality risk score is corrected to obtain the comprehensive quality score.

[0016] A scientific research data quality verification system based on AI machine learning includes: a hierarchical structure construction unit for collecting heterogeneous scientific research data, constructing a hierarchical data structure through semantic knowledge base construction and automatic field mounting, including a patient layer, visit layer, form layer, and field layer; a dependency relationship mining unit for performing dependency relationship mining on the hierarchical data, constructing a dependency relationship graph containing a set of edges, including dependency constraint edges and logical edges; a hierarchical heterogeneity identification unit for identifying anomalies based on the dependency relationship graph and hierarchical data using a pre-built hierarchical heterogeneity identification model, generating rule violation residuals, time series prediction residuals, cross-center alignment residuals, and topology reconstruction residuals; a comprehensive quality scoring unit for comprehensively processing each residual, generating a comprehensive quality score, and automatically classifying the problem level; and an evidence chain storage unit for encrypting and storing evidence chains for problems that meet the preset level, and generating an audit report.

[0017] The present invention has the following beneficial effects:

[0018] (1) This AI machine learning-based scientific research data quality verification method builds a hierarchical data structure of patient layer, visit layer, form layer and field layer, and combines a semantic knowledge base and automatic field mounting method. It can unify and organize heterogeneous scientific research data from different sources and in different formats into a standardized hierarchical framework. At the same time, it relies on the dependency graph to associate scattered fields into an overall topological structure, avoiding the limitation of verifying only a single field. It can carry out cross-dimensional verification according to the constraints of time sequence, field logic, form association, etc., thereby discovering hidden data anomalies hidden between cross fields, cross visits and cross forms, reducing the situation of missing problems in manual investigation, and thus improving the accuracy of scientific research data quality verification.

[0019] (2) This AI machine learning-based scientific research data quality verification method constructs a dependency graph and combines it with a hierarchical heterogeneous identification model to generate four types of residual joint judgment. That is, the time sequence, logical relationship and form constraints between fields are sorted out and formed. Then, through rule verification, time series trend prediction, adversarial domain alignment detection and graph topology reconstruction, the degree of deviation of data from the normal pattern is calculated from different dimensions to form a multi-dimensional residual index. Combined with the data's own correlation pattern, visit change trend and multi-center distribution characteristics, the method can accurately find hidden data problems, reduce the situation of missed judgment and misjudgment, and make the data problem investigation more comprehensive and detailed.

[0020] (3) The scientific research data quality verification method based on AI machine learning is to perform adaptive weighted calculation by integrating four types of residuals, and then combine the important weight of fields and the evidence chain integrity coefficient correction calculation to obtain a comprehensive quality score. The method automatically classifies the problem level according to the score, and completes the evidence chain encryption and storage for problems that meet the requirements. This avoids the shortcomings of not being able to distinguish the main and secondary problems and lacking traceability vouchers. After completing the evidence chain encryption and storage, an audit report can also be generated, so that data anomalies can retain complete and traceable records, which is suitable for the actual business scenarios of daily verification and compliance traceability of scientific research projects.

[0021] (4) This AI machine learning-based scientific research data quality verification system achieves standardized hierarchical organization and semantic uniform mounting of multi-source heterogeneous scientific research data through hierarchical structure construction unit, automatic construction of field association topology and logical constraint sorting of dependency mining unit, residual quantitative calculation of various implicit and explicit anomalies from multiple dimensions based on multi-model architecture, adaptive fusion of multiple residuals and intelligent classification of problem levels by comprehensive quality scoring unit, automatic collection of high-level anomaly information, encrypted storage and autonomous generation of audit reports by evidence chain storage unit, and cooperation between the units to achieve intelligent operation of scientific research data quality control process, streamline manual operation links, and thus effectively improve the overall operation efficiency and process standardization of scientific research data quality verification.

[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0023] Figure 1 This is a flowchart of a scientific research data quality verification method based on AI machine learning, according to the present invention.

[0024] Figure 2 This is a flowchart illustrating the specific steps involved in generating cross-center aligned residuals in a scientific research data quality verification method based on AI machine learning, as described in this invention.

[0025] Figure 3This is a block diagram of a scientific research data quality verification system based on AI machine learning, according to the present invention. Detailed Implementation

[0026] Please see Figure 1 This invention provides a technical solution: a method for verifying the quality of scientific research data based on AI machine learning, comprising the following steps: collecting heterogeneous scientific research data, constructing a semantic knowledge base and automatically mounting fields to build a hierarchical data structure, which includes a patient layer, a visit layer, a form layer, and a field layer, wherein the field layer is used to store the value of each field and its attribute information, and the attribute information includes the field name, field value, modification record, data source, and field importance level; the patient layer at least includes the patient identifier and the research center to which the patient belongs; performing dependency mining on the hierarchical data structure to construct a dependency graph containing an edge set, the edge set including dependency constraint edges and logical edges, each edge being used to constrain the dependency relationship between different field values;

[0027] Based on dependency graphs and hierarchical data, a pre-built hierarchical heterogeneous identification model is used for anomaly identification, including: a rule validator, which traverses the constraint edges in the dependency graph and performs violation verification to generate rule violation residuals; a longitudinal trend predictor, which statistically analyzes the longitudinal sequence of visits based on the dependency graph and hierarchical data, and generates time-series prediction residuals through cross-visit temporal attention encoding and trend extrapolation; a domain alignment detector, which extracts a distribution-consistent field set based on the dependency graph and hierarchical data, and calculates reconstruction error and dynamic adaptive threshold correction through adversarial training to generate cross-center alignment residuals; and a dependency graph detector, which constructs a heterogeneous graph based on hierarchical data and dependency graphs, and generates topology reconstruction residuals through graph structure propagation and iterative reconstruction.

[0028] The residuals are comprehensively processed to generate a comprehensive quality score, and the problem levels are automatically classified. A query record containing trigger rules and suggested values ​​is generated, specifically as follows:

[0029] Preset four problem level thresholds to achieve precise classification:

[0030] Minor issues: The data deviation is minimal and does not affect the overall validity of the research data; it can be reviewed in batches on a regular basis.

[0031] General questions: There are minor data anomalies that require routine verification and correction.

[0032] Important issues: There are significant data discrepancies that may affect the results of local data statistics, and these need to be rectified as a priority.

[0033] Serious problem: There is a significant risk of data errors or falsification, which seriously affects the authenticity and reliability of research data. The data needs to be verified and locked immediately.

[0034] Based on the comprehensive quality score, each suspicious data point is classified into the corresponding problem level. For abnormal data that reaches the "general problem" level or above, the system automatically generates structured records, captures core information to construct standardized question items, which specifically include the patient's unique identifier, the research center to which the patient belongs, visit node information, form name, target field name, original field value, abnormal trigger source (preset rule / AI model), corresponding residual type and specific value, and intelligent correction suggestion value.

[0035] The recommended value for numerical fields is the optimal prediction value of the time series prediction model, and the recommended value for categorical fields is the high-frequency association compliance category value obtained from dependency mining. For "minor issues", only brief logs are recorded for periodic batch review, and no formal questioning records are generated.

[0036] All challenge records are stored in a dedicated challenge database in a standardized JSON format, with a reserved evidence_hash field initially empty. For serious and important anomalies, the data is automatically pushed to the quality control platform. It also supports two-way push of warning information via email and API interface to notify the corresponding research center's quality control personnel to complete the review and correction.

[0037] For issues that meet the preset level, the evidence chain is encrypted and stored, and an audit report is generated, specifically as follows:

[0038] The system filters out serious and important anomalies (general anomalies can be selected by the user). It automatically collects evidence from all dimensions to build a complete chain of evidence. The evidence includes: anomaly location information (patient ID, research center, visit / form / field information), original field values ​​and full version modification history (including operator and timestamp), anomaly triggering rules and model parameters, complete residual calculation process, one-hop dependency data of related fields, model version and training snapshot identifier.

[0039] A chain hash encryption mechanism is used to complete the evidence storage. Every 1,000 pieces of evidence are accumulated or at a fixed time each day, the hash value at the end of the chain is synchronized to a third-party authoritative evidence storage platform to obtain a timestamp for solidification, which prevents evidence tampering and time forgery. After the evidence storage is completed, the final hash value is filled back into the `evidence_hash` field of the corresponding question record to complete the evidence loop.

[0040] Compliance audit reports are automatically generated based on solidified evidence chains, including but not limited to: overall statistics on data quality anomalies (number of issues at each level, central distribution, and field distribution), complete evidence tracing information for each anomaly, quality scoring and level determination criteria, intelligent correction suggestions, and issue handling status (pending / responded / resolved).

[0041] The patient layer is used to store the main information of an individual patient, including at least the patient's unique identifier and the research center to which they belong.

[0042] The visit layer is used to store the patient's visit records at each visit time point. Each visit node includes the planned visit date, the actual visit date, and the visit completion status, and establishes a subordinate relationship with its corresponding patient.

[0043] The form layer is used to store the Case Report Form (CRF) instances filled out under each visit. Each form records its form type (such as demographic form, vital signs form, adverse event form) and establishes a subordinate relationship with the visit it belongs to.

[0044] The field layer stores the specific values ​​and attribute information of each field in each form, and is the finest-grained level of this hierarchy.

[0045] Specifically, the steps for constructing hierarchical data are as follows:

[0046] Heterogeneous research data is extracted to a temporary storage area via an ETL process. Data tables are then retrieved from the temporary storage area, and each table is categorized according to a predefined hierarchical mapping rule (the hierarchical mapping rule specifies how to categorize columns in the data table into four categories: patient identifier, visit node, form type, and field). Specifically:

[0047] Heterogeneous research data sources include EDC systems, electronic medical records, Excel files, and SDTM datasets that conform to the CDISC standard. The ETL process extracts data from each data source and stores the raw data in its original format into a temporary storage area. The temporary storage area adopts a columnar storage structure to retain the complete modification history and timestamps of the data.

[0048] Metadata information of each data table is obtained from the temporary storage area, including table name, column name, data type and constraints. The preset hierarchical mapping rules are stored in the form of JSON configuration files. Each rule contains three elements: the regular expression matching pattern of the source data table / column, the target hierarchical level (patient level / visit level / form level / field level), and the mapping priority (1-10, the smaller the number, the higher the priority).

[0049] In one implementation, the mapping rules are as follows: the matching pattern "subject number" is mapped to the patient identifier field in the patient layer, with a priority of 1; the matching pattern "visit name" is mapped to the visit node name field in the visit layer, with a priority of 1; the matching pattern "adverse event" is mapped to the form type "adverse event table" in the form layer, with a priority of 2. The mapping is executed in descending order of priority. If a column matches multiple rules at the same time, the mapping result with the highest priority is adopted, and the mapping conflict information is recorded in the log for manual review.

[0050] The categorized data tables are organized into an initial structure (the initial structure includes an initial patient layer, an initial visit layer, an initial form layer, and an initial field layer, where the initial field layer temporarily stores the values ​​and basic attributes of each original field). A semantic knowledge base containing standard domain concepts, field alias mappings, unit conversion rules, and default quality control rules is pre-built and stored in a graph database. Specifically:

[0051] The initial structure includes an initial patient layer, an initial visit layer, an initial form layer, and an initial field layer. The initial field layer temporarily stores the values ​​and basic attributes (field name, data type, original unit, collection time, and data entry personnel) of each original field. The fields in the initial structure are not yet associated with standard concepts, and the field names may vary depending on the research project. For example, systolic blood pressure may be named "SBP", "systolic blood pressure", "systolic_blood_pressure", or "high pressure".

[0052] The semantic knowledge base is stored in the Neo4j graph database, and its node types include: semantic concept nodes (such as "systolic blood pressure"), field alias nodes (such as "SBP"), form type nodes (such as "vital signs table"), visit type nodes (such as "baseline visit"), and quality control rule nodes (such as "systolic blood pressure range 90-140 mmHg").

[0053] Nodes are connected by relation edges, with relation types including: HAS_ALIAS (with aliases), BELONGS_TO_FORM (belonging to forms), HAS_RULE (with rules), and REQUIRES_UNIT_CONVERSION (requiring unit conversion). The semantic knowledge base pre-includes 2000+ common clinical concepts, 5000+ field aliases, 100+ unit conversion rules (such as the conversion factor of 7.5 between mmHg and kPa, and the conversion factor of 0.0555 between mg / dL and mmol / L), and 300+ default quality control rules.

[0054] For each original field in the initial field layer of the initial structure, its value and basic attributes are vectorized, and semantic matching is performed with standard concepts in the semantic knowledge base using semantic similarity calculation. Specifically:

[0055] Text preprocessing is performed on the field names of the original fields, including lowercase conversion, removal of non-alphanumeric characters, word segmentation, and stemming. The pre-trained BioBERT model (a pre-trained language model in the biomedical field) is used to encode the pre-processed field names into semantic vectors. The BioBERT model adopts the PubMedBERT-base-uncased-abstract-fulltext version and is fine-tuned on positive and negative sample pairs composed of standard clinical terminology pairs (such as MedDRA terms and common aliases). The loss function adopts the contrastive loss to maximize the cosine similarity of matching field name-concept name vectors and minimize the similarity of unmatched vectors.

[0056] Simultaneously, the name of each semantic concept node in the semantic knowledge base and the names of all associated alias nodes are encoded into vectors of the same dimension, and semantic similarity is calculated. Cosine similarity is used for calculation, and the formula is: ;

[0057] in, A semantic vector for the original field name. For semantic concept nodes, a name vector or alias vector is used. This represents the dot product operation of vectors. and These represent the magnitudes of the two vectors, with values ​​closer to 1 indicating closer semantic similarity.

[0058] For each original field, the semantic concept with the highest similarity is selected as the matching result. If the highest similarity is lower than the preset threshold of 0.7, it is marked as "to be manually mapped", and all candidate concepts with similarity greater than 0.5 are recorded for manual selection.

[0059] In one implementation, the semantic similarity between the original field "SBP" and the semantic concept "systolic blood pressure" is 0.94, which is a successful match; the similarity between "high blood pressure" and "systolic blood pressure" is 0.88, which is also a successful match; while the similarity between "blood pressure notes" and "systolic blood pressure" is 0.45, which is below the threshold and requires manual intervention.

[0060] Based on the semantic matching results, semantic attachment and rule inheritance are performed, and the data is backfilled into the initial structure to build a hierarchical data structure, specifically as follows:

[0061] For each successfully matched original field, establish a mapping relationship from the original field to the semantic concept node and store it in the mapping table;

[0062] Inherit the default quality control rules associated with the semantic concept node from the semantic knowledge base. For example, the rules associated with the "systolic blood pressure" node include: range rule (systolic blood pressure should be between 90-140 mmHg), trend rule (the change in systolic blood pressure between adjacent visits should not exceed 40 mmHg), and unit conversion rule (if the original unit is kPa, it will be automatically converted to mmHg).

[0063] If the original field carries unit information and is inconsistent with the default unit of the semantic concept, the unit conversion is automatically performed to convert the field value into the standard unit and then store it in the field layer;

[0064] The backfill operation writes the semantic concept name, the list of inherited rules, and the converted field value into the field layer of the initial structure, while retaining the original field name and original value as audit trail information.

[0065] For fields marked "to be manually mapped", the system generates a confirmation task in the quality control interface. Researchers manually select semantic concepts or create new semantic concept nodes. After manual confirmation, the mapping is completed. After all fields are mapped, the initial structure is upgraded to a complete hierarchical data structure. Each field in the field layer includes: logical field name (standard semantic concept), original field name, field value (standardized), unit (standard unit), modification record (including version number and timestamp), data source, inheritance rule list, and field importance level (inherited from semantic concept or manually set).

[0066] Field importance is divided into four levels: primary endpoint fields (weight 1.0), critical security fields (weight 0.9), core inbound and outbound criteria fields (weight 0.8), and general auxiliary fields (weight 0.5).

[0067] The specific steps for constructing a dependency graph are as follows:

[0068] Extract several fields from the field layer of the hierarchical data and use them as field nodes, specifically:

[0069] Extract all fields from the field layer of the hierarchical data. Each field corresponds to a field node. The attributes of the field node include: node identifier (formatted as "Patient ID_Visit Node_Form Type_Logical Field Name"), logical field name, field importance level, data type (numeric / categorical / date), research center and form type.

[0070] For the same logical field that appears repeatedly during the same visit for the same patient (e.g., multiple measurements of the same indicator in laboratory tests), the earliest value is taken as the master record based on the collection timestamp, and the remaining records are stored as secondary records without creating separate nodes to avoid graph structure bloat. The total number of field nodes is denoted as […]. The range can be set to 50-500;

[0071] Add dependency constraint edges between field nodes that meet preset logical constraints. These logical constraints include time sequence dependencies, field logical dependencies, and inter-form consistency dependencies, specifically:

[0072] Logical constraints are derived from the medical logical relationships explicitly defined in the research protocol and are pre-set in the constraint rule base. The constraint rule base contains three types of constraints: time order dependency, field logical dependency, and inter-form consistency dependency.

[0073] For time-order dependencies, such as "enrollment date should be earlier than the first dose date," find the "enrollment date" field node in the patient layer and the "dose date" field node in the form layer's dosing record table. Add a directed dependency constraint edge between them, pointing from "enrollment date" to "dose date." The edge's attributes include: constraint type "time order," constraint expression... The violation tolerance threshold is 0 days (i.e., violations are not allowed);

[0074] For example, to establish a logical dependency between fields, such as "If the 'Pregnant Woman' field is 'Yes', then the 'Pregnancy Test' field must be 'Positive'", find the "Pregnant Woman" field node and the "Pregnancy Test Result" field node, and add a bidirectional logical dependency edge with the following attributes: constraint type "logical implication" and constraint expression. ;

[0075] For example, to establish consistent dependencies between forms, such as "the evaluation conclusion in the efficacy evaluation table should be consistent with the RECIST conclusion in the imaging evaluation table," find the corresponding field nodes in both forms and add a consistency dependency edge with the following attributes: constraint type "consistency" and constraint expression. All dependent constraint edges are accompanied by constraint threshold parameters. When the actual deviation between field values ​​exceeds the threshold, an anomaly flag is triggered.

[0076] Data cleaning and pre-screening are performed on hierarchical data to generate qualified research data, specifically as follows:

[0077] Data cleaning removes records where the field value is empty and the field is marked as a "key field" (field importance level ≥ 0.8), and removes records with obvious format errors, such as date fields that cannot be parsed into valid dates, numeric fields containing non-numeric characters, and category fields whose values ​​are not within the preset enumeration range;

[0078] Univariate statistical outlier removal: For each (research center, logical field) combination, calculate the mean and standard deviation of the field values, and remove outliers. For extreme values ​​within the range, if the overall missing rate of a research center exceeds 50% or the overall visit out-of-window rate exceeds 80%, all data from that research center will be temporarily marked as "low quality" and will not be used for subsequent dependency mining. However, a separate data quality report will be retained for manual review, and the cleaned dataset will be recorded as qualified research data. ;

[0079] Based on qualified scientific research data, statistical dependency mining and causal structure learning methods are used for different fields to generate logical edges. Each logical edge is accompanied by a comprehensive confidence score output by causal inference, which is as follows:

[0080] Dependency mining can use the Apriori association rule mining algorithm, with a minimum support of 0.05 (i.e., the frequency of field value combinations is not less than 5%) and a minimum rule strength of 0.7 (i.e., the conditional probability is not less than 70%).

[0081] By mining all categorical fields and discretized numerical fields in qualified scientific research data, we discovered frequently occurring combinations of field values.

[0082] For example, if the probability of "ALT increase" and "AST increase" occurring simultaneously is found to be 95%, the support is 0.32, and the rule strength is 0.95, then a statistical dependency edge is added between the "ALT" field node and the "AST" field node. The edge type is "statistically related (candidate)" and the rule strength is 0.95.

[0083] Causal structure learning can be achieved using the PC algorithm (Peter-Clark algorithm), which constructs a directed acyclic graph based on the conditional independence test. The conditional independence test uses the partial correlation coefficient test with a significance level of 0.01. It first constructs a completely undirected graph, then deletes edges through successive conditional independence tests, and finally determines the direction of the edges based on the V-shaped structure recognition rule and the Meek orientation rule.

[0084] The discovered directed edges were subjected to intervention testing to improve the causal confidence: the intervention test used propensity score matching, taking a certain field value as the "intervention", and then testing the significance of the causal effect after matching with other covariates;

[0085] The specific method for intervention testing is as follows: the initial field values ​​of the candidate edges are binarized into high / low levels according to clinical thresholds, and these are used as intervention variables. Age, gender, baseline values, and center identifiers are used as covariates. Logistic regression is used to estimate propensity score and 1:1 nearest neighbor matching is performed. Then, the average intervention effect is estimated through a dual machine learning method (using random forest as the base learner and eliminating the influence of redundant parameters by orthogonalizing the score function). If the 95% confidence interval of the effect estimate does not contain 0 and p < 0.05, the causal direction is confirmed.

[0086] If the p-value of the independence test after propensity score matching is <0.05, the causal direction is confirmed, and the edge type is upgraded to "Statistically Related (Confirmed)". Otherwise, it remains "Statistically Related (Experimental)". Each logical edge is accompanied by a comprehensive confidence score. The calculation formula is: ;

[0087] in, This represents the conditional probability of the association rule. The p-value is the result of the conditional independence test. Intervention tests are marked as 1 (if passed) or 0 (if not passed). The closer the confidence level is to 1, the more reliable the dependency is.

[0088] And assign distribution consistency tags to field nodes that meet the preset cross-center consistency conditions, specifically as follows:

[0089] Cross-center consistency criteria are used to label fields that are theoretically unaffected by center differences, such as demographic characteristic fields like “height”, “date of birth”, and “gender”, as well as standardized coded fields like “adverse event outcome” and “name of combined medication”.

[0090] The system matches the logical field names of the fields against a pre-defined whitelist of consistent fields. If a match is found, a consistent flag is assigned. ,otherwise Furthermore, the consistent distribution markers are used in subsequent domain alignment detectors to determine which fields participate in adversarial training. The distribution differences of fields marked as 1 between different centers are theoretically only due to differences in data quality rather than real medical differences, and are an effective reference for identifying center shifts.

[0091] The constraint edges, logical edges, and field nodes are combined into a dependency graph, which is as follows:

[0092] Dependency graph with directed weighted graph Storage, in For a collection of field nodes, The set of edges contains two types: constraint edges and logical edges. Constraint edges are derived from manually preset logical constraints (such as time order dependencies, field logical dependencies, and consistency dependencies between forms); logical edges are automatically mined by AI (based on association rules and causal structure learning) and are further subdivided into three confidence states: "candidate", "confirmed" and "experimental" according to the causal intervention test results.

[0093] Each edge Storage attribute quadruple: , representing edge type, constraint threshold, overall confidence level, and creation timestamp, respectively, and the total number of nodes in the dependency graph. Total number of sides It is typically 1.5 to 3 times the number of nodes.

[0094] Specifically, when the amount of new data or changes in the graph structure exceed a preset threshold, dependency mining is re-executed, updating the weights of existing edges, adding newly discovered high-confidence dependency edges, reducing the weights of edges that have not been violated for a long time or have low confidence, and saving a version snapshot to support the tracking of evolution history.

[0095] The threshold condition for triggering re-dig is:

[0096] The number of newly added qualified data records exceeded 200;

[0097] The number of edges in the graph structure changes by more than 10%;

[0098] More than 30 days have passed since the last excavation;

[0099] When any condition is met, the incremental mining process is automatically initiated. Incremental mining reuses historical mining results, performing statistical dependency and causal structure learning only on new data. Newly discovered dependency edges are merged with historical edges, following these rules: if a new edge and a historical edge connect to the same pair of nodes, the updated confidence score is the weighted average of the two, with the weight proportional to the amount of data. ;

[0100] in, This represents the number of historical data records. To add the number of data records, , These are historical confidence levels and new confidence levels, respectively.

[0101] If a new edge connects a node pair that does not exist in the historical graph, it is directly added as a new logical edge with an initial type of "statistical relevance (experimental)". For statistically relevance edges that have not been violated for a long period (more than 90 days) and whose data volume supports the statistics, their weight is reduced by 10%.

[0102] If the edge is reduced in three consecutive updates and the confidence level is below 0.3, then mark the edge as "obsolete" and remove it from the graph (retain historical versions for querying).

[0103] After each dependency graph update, a complete snapshot of the graph structure is saved, including all nodes, edges, and attributes.

[0104] In this implementation plan, the semantic vectorization matching of fields is completed by relying on the pre-trained model, the standard concepts are automatically associated and various business and conversion rules are inherited, and the heterogeneous fields are normalized and integrated. At the same time, the logical pre-set constraints between fields are combined, and statistical and causal relationships are autonomously mined from compliant historical data to dynamically generate field topology dependency networks. The dependency structure can also be continuously iterated and updated as data accumulates and version records are retained, thereby realizing data standardization and in-depth mining of implicit relationships.

[0105] Specifically, the steps involved in generating rules that violate residuals are as follows:

[0106] The constraint type and corresponding constraint threshold of the constraint edge are determined as follows:

[0107] Extract the attributes of each constraint edge from the dependency graph. The constraint types include: time order constraints (e.g., "A is earlier than B"), numerical range constraints (e.g., "A is in the range [L, U]"), logical implication constraints (e.g., "If A=true, then B=true"), consistency constraints (e.g., "A=B"), and enumeration value constraints (e.g., "A∈{value1, value2, ...}").

[0108] Each constrained edge is accompanied by a constraint threshold parameter. Regarding time sequence constraints, This indicates the maximum number of days allowed in advance (usually 0); for numerical range constraints, This indicates the maximum permissible deviation (in standard deviation, typically 3); for logical implication constraints, The value is fixed at 0 (violation is not allowed); for consistency constraints, Indicates the maximum allowed difference (numerical) or 0 (categorical);

[0109] Based on the dependency graph, the set of associated field nodes that have constraint edges with the current field node is extracted, specifically as follows:

[0110] For a given target field node The system performs a breadth-first search in the dependency graph, with a depth limit of 1, to find all nodes connected to the system through constraint edges. Directly connected field nodes, these associated field nodes and There must be at least one constraint edge between them, and the direction of the edge may be from the associated node to the target node, from the target node to the associated node, or bidirectional.

[0111] The set of associated nodes is denoted as ,in This represents the number of associated nodes;

[0112] In one implementation example, the associated nodes of the target field node "Drug Administration Date" may include "Enrollment Date" (time order constraint: Enrollment Date ≤ Drug Administration Date) and "Efficacy Evaluation Date" (time order constraint: Drug Administration Date ≤ Efficacy Evaluation Date);

[0113] Determine whether a field node and its corresponding associated field node set satisfy the constraint type and corresponding constraint threshold of the constraint edge. Specifically:

[0114] Traverse each line from point to Or from point to Given the constraint edges, obtain the corresponding field values ​​based on the edge direction, and set the field nodes. The actual value is Related field nodes The actual value is Perform a judgment based on the constraint type:

[0115] Time sequence constraints: verification (If the direction is) Earlier )or (If the direction is) Earlier );

[0116] Numerical range constraints: verification ,in , These are the lower and upper bounds of the normal range. This is the historical standard deviation of this field;

[0117] Logical implication constraint: If If condition P is satisfied, then verify Condition Q must be met;

[0118] Consistency Constraints: Verification (Numerical type) or (Classification);

[0119] If not satisfied, constraint deviation mapping is performed on the field node, its corresponding associated field node set, and the constraint threshold to generate rule violation residuals, specifically as follows:

[0120] For time order constraints, if ,but ,in The maximum tolerable window number (e.g., 14 days) is set as the preset maximum tolerance window number, and the denominator is used to map the window number to... The range, exceeding The residual is truncated to 1;

[0121] For numerical range constraints, if ,but ;like ,but The denominator 3σ indicates that the maximum residual is reached when the deviation is 3 standard deviations.

[0122] For logical implication constraints, if violated, then (Binary residuals, no intermediate states);

[0123] For consistency constraints, if numerical consistency is violated, then ,in The maximum allowable difference is preset (e.g., 10 units).

[0124] It should be noted that if the same target field node violates multiple constraint edges at the same time, the maximum value of the residuals calculated from each edge is taken as the final rule violation residual of that field node.

[0125] If satisfied, the rule violation residual is set to 0;

[0126] In one implementation, a patient's dosing date was January 10, 2025, and their enrollment date was January 15, 2025. The enrollment date was 5 days later than the dosing date, violating the time order constraint of "enrollment date ≤ dosing date". sky, Heaven, then .

[0127] In this implementation plan, various constraint relationships between fields are automatically matched based on the dependency graph. Dedicated judgment logic is adapted according to different constraint types. By accurately comparing the actual values ​​of fields with constraint thresholds, the degree of violation deviation is quantified and normalized to generate rule violation residuals. For multiple constraint violations, the best value can be selected to unify the judgment standard, thereby achieving a refined measurement of the degree of data rule violation.

[0128] Specifically, the steps for generating time-series prediction residuals are as follows:

[0129] Based on the field nodes in the dependency graph and the visit and field layers in the hierarchical data, the target fields for vertical trend prediction are determined, specifically:

[0130] Target fields for longitudinal trend prediction include: laboratory test indicators (such as ALT, AST, creatinine, white blood cell count), vital signs indicators (such as systolic blood pressure, diastolic blood pressure, heart rate, body temperature), and efficacy evaluation indicators (such as total tumor diameter, CD4 count, viral load).

[0131] The field is matched against a pre-defined longitudinal prediction whitelist based on its logical field name. Fields in the whitelist are usually continuous numerical variables that are expected to show a trend of change over time during multiple visits. If a match is successful, the field is marked as the target field for longitudinal prediction. Non-continuous variables (such as categorical variables) or fields that should theoretically not have a trend of change between visits (such as date of birth and gender) are not included in the longitudinal prediction.

[0132] For each target field under each patient identifier, the longitudinal sequence of visits is extracted in chronological order of visit time, specifically as follows:

[0133] For patients and target field Extract all visit node records for the patient from the hierarchical data, sort them in ascending order by actual visit date (unplanned date), and process each visit... ( , (Total number of visits for this patient), extract the target field under this visit. field values and the actual time interval since the last visit (Number of days), first visit (baseline visit) The longitudinal sequence of visits is defined as: ;

[0134] in For the first The field value for each visit; if the field is missing for a particular visit, then... If a position is marked as NaN, it will be retained in the sequence, but when used as model input, the Last Observation Carryover (LOCF) method will be preferentially used for forward imputation. If more than two consecutive missing positions are found, the sequence will be marked as "low quality" and the weight of the time series prediction residuals will be reduced in the subsequent comprehensive score. The sequence length... It varies from patient to patient and depends on the actual number of visits performed. (Baseline visit only) The time series prediction residuals are set directly to 0;

[0135] The longitudinal sequence of visits is input into the time-series prediction model. The time-series prediction model uses cross-visit time-series attention encoding based on actual time intervals to perform weighted aggregation of field values ​​and time intervals for each visit node in the longitudinal sequence of visits, thereby obtaining the predicted field value and field prediction confidence interval for the current visit. Specifically:

[0136] The time series prediction model employs a two-layer LSTM (Long Short-Term Memory) network structure, with the hidden layer dimension set to 64 and the input layer accepting lengths of [missing information]. The sequence, the feature vector of each time step is (Notice Using only historical values, at the current time step (The target to be predicted is not input into the model).

[0137] The model training data is qualified scientific research data. The training objective is to minimize the Huber loss between predicted and actual values ​​for all patients' longitudinal sequences (the threshold parameter can be set to 1). A cross-visit temporal attention mechanism adds an attention layer on top of the LSTM output, calculating the contribution weight of each historical visit to the prediction of the current visit, using the following formula: ;

[0138] in, For the first The LSTM hidden state at each time step , The attention weight matrix is ​​a learnable attention mechanism that enables the model to automatically focus on the preceding visit nodes that are most relevant to the current visit prediction. For example, the visit closest to the current visit has a higher weight, or the visit with a significant trend of change has a higher weight.

[0139] The model outputs the predicted values ​​of the fields for the current visit. and the standard deviation of prediction The prediction confidence interval is set to (Corresponding to a 95% confidence level);

[0140] In one implementation, the predicted sequence of a patient's ALT value is: baseline visit ALT = 35 U / L ( During the first treatment visit, ALT = 42 U / L. Day), during the second treatment visit, ALT = 50 U / L ( Day), model predicts 3rd treatment visit ( The ALT value for (day) was 58 U / L, the predictive standard deviation was 5 U / L, and the 95% confidence interval was 48-68 U / L.

[0141] From the hierarchical data, the current visit field value of the target field is extracted, and confidence residual mapping is performed with the field prediction value and the field prediction confidence interval to generate time series prediction residuals. Specifically:

[0142] Let the actual field value of the current visit be... Time series prediction residuals The calculation formula is: ;

[0143] in, The confidence interval is half-width;

[0144] In the aforementioned implementation, the actual ALT value is 85 U / L, the predicted value is 58 U / L, and the confidence interval half-width is 10 U / L. , This indicates that the value deviates significantly from the time-series forecast trend, triggering a high time-series forecast residual;

[0145] If the field value of the current visit is missing, the time series prediction residual is set directly to 1 (severe anomaly). If the patient's historical longitudinal sequence length is less than 2 (i.e. only the baseline visit is present), the time series prediction residual is set to 0.

[0146] like Figure 2 As shown, the specific steps for generating cross-center aligned residuals are as follows:

[0147] Based on the field nodes marked as having consistent distribution, a consistent set of fields is extracted from the hierarchical data, and then grouped according to the research center to which the field corresponding to each field node belongs. Specifically:

[0148] Filter out markers with consistent distribution from the dependency graph. The field nodes, whose logical field names belong to a pre-defined consistent field whitelist, include: height, date of birth, gender, adverse event code (such as MedDRA code), combined medication code (such as ATC code), etc.

[0149] Extract the field values ​​of these field nodes from the hierarchical data to form a consistent field set. Let there be a total of Each research center Extract all patients from the center The field values ​​above form a grouped dataset. ,in This represents the total number of records at the center. For the first The field values ​​of the record. This serves as the identifier for the visited node. For form type identification;

[0150] For the grouping results, historical research data are extracted from the hierarchical data, and a dynamic adaptive threshold is generated through kernel density estimation, specifically as follows:

[0151] For each research center Consistent field for each distribution Use the center's historical qualified data (from Kernel density estimation is performed to obtain field values. probability density function ;

[0152] Kernel density estimation uses a Gaussian kernel, and the bandwidth is selected according to the Silverman rule: ,in, The standard deviation of the field value for that center. For sample size;

[0153] Calculate the 95% confidence interval from the probability density function. , making For the new center (sample size) The Bayesian hierarchical model is adopted: the global distribution (mean μg, variance σg²) is calculated by merging the data of all centers with a sample size ≥30 as the prior; if there are fewer than 3 centers with a sample size ≥30, the general distribution parameters of the field are obtained from the physiological indicator prior library shared across projects as the prior, and the global distribution is assumed to follow a normal distribution. Data from the new center ( The variance estimate for a single observation is: ;

[0154] Based on the normal-normal conjugate model, the posterior mean and variance are as follows: ;

[0155] in Given the sample mean of the new center, calculate the dynamic threshold interval of the new center based on its post-abortive distribution. Take the 95% highest density interval of the posterior distribution, and use dynamic threshold rule residuals. The calculation is as follows: ;

[0156] The denominator is the half-width of the threshold interval. When the field value exceeds the threshold boundary, the residual is positive; when the distance exceeds the threshold reaches the half-width of the interval, the residual is 1.

[0157] An adversarial domain adaptation network is constructed, comprising a feature extractor, a center discriminator, and a reconstructionist. Through adversarial training, feature representations independent of the center are learned, and the reconstruction error is calculated to obtain the domain alignment residual. Specifically:

[0158] Adversarial domain adaptation networks consist of three parts: feature extractor Central discriminator and refactor ;

[0159] The feature extractor uses a three-layer fully connected network, with the number of nodes in each layer set to 128, 64, or 32 respectively. The activation function is ReLU, and the input is field values. Its contextual features (normalized visit number, form type code, normalized patient age, gender code), concatenated into an input dimension denoted as... The output is the latent space feature vector. ,in The input feature vector;

[0160] The center discriminator uses a two-layer fully connected network, and its output is center classification logits. The activation function is Softmax.

[0161] Reconfigurator Employing a self-encoder structure: the encoder will Mapping back to 32 dimensions, the decoder maps the 32 dimensions back to... Dimension, output reconstructed input ;

[0162] Domain Alignment Residual Defined as input feature vector Rather than reconstruct Squared Euclidean distance between them: ;

[0163] The smaller the value, the better the data point can be reconstructed, conforming to the distribution pattern learned by the model from normal data; the larger the value, the more the data point deviates from the normal pattern and may be an anomaly.

[0164] The dynamic threshold residual and the domain alignment residual are jointly corrected, and the result of the joint correction is used as the cross-center alignment residual. Specifically:

[0165] Cross-center aligned residuals The joint correction formula is: ;

[0166] in and These are weighting coefficients, with initial values ​​set to 0.4 and 0.6 respectively, which can be dynamically adjusted based on the quality of the central data. The normalized threshold for the domain alignment residuals is taken from historical normal data. The 95th percentile;

[0167] according to and The joint judgment rules output different exception handling actions:

[0168] like and If it is identified as "suspicious anomaly (outlier at the center but common globally)", the cross-center alignment residual is multiplied by a coefficient of 0.5.

[0169] like and The error was identified as "abnormal real data (isolated error)", and the cross-center aligned residual proceeded normally to the subsequent scoring.

[0170] If both exceed the limits, it is judged as "real anomaly + possible center shift", and the priority of cross-center alignment residuals is increased (multiplied by a coefficient of 1.2 in subsequent scoring).

[0171] If neither exceeds the limit, it is considered normal. .

[0172] The construction steps of the adversarial domain adaptation network are as follows:

[0173] The values ​​of each field in the grouping results, along with their respective visit nodes and form types, are input into the feature extractor. After deep feature encoding, the latent space feature vector is output, specifically:

[0174] Constructing input feature vectors Its components are: field values (For numeric fields, use the standardized value directly. The standardization formula is:) ,in , The mean and standard deviation of this field in the global training set; the categorical field is constructed by concatenating one-hot encoding, the normalized value of the visit node number (actual visit order / maximum number of visits), the form type encoding (one-hot, dimension is the total number of form types), the normalized value of the patient age (age / 120), and the gender encoding (0 / 1).

[0175] All components are concatenated to form the input vector. Feature extractor It is a three-layer fully connected network: ;

[0176] in, , , For a trainable weight matrix, Given a trainable bias vector, the ReLU activation function is defined as follows: Output latent space feature vectors ;

[0177] The latent space feature vectors are simultaneously input into the center discriminator and the reconstructor, specifically as follows:

[0178] Latent space feature vectors It is copied into two copies and sent to the central discriminator separately. and refactor The center discriminator is used to predict the research center to which the input data belongs, and the reconstructor is used to reconstruct the original input from the latent space. Both share the output of the same feature extractor.

[0179] The center discriminator predicts the center label to which the latent space feature vector belongs and calculates the classification loss; the reconstructor reconstructs the latent space feature vector, outputs the reconstructed feature vector, and calculates the reconstruction loss, specifically as follows:

[0180] Central discriminator It is a two-layer fully connected network: , The output dimension is the number of research centers. The central probability distribution is then obtained by following the Softmax function. The classification loss uses cross-entropy loss: ;

[0181] in For batch size, For the first One-hot encoding of the true center label of each sample (1 indicates belonging to the center). (0 indicates not to be included). The model predicts the first Each sample belongs to the center The probability of;

[0182] Reconfigurator For autoencoders: encoding part (Output 32-dimensional), Decoding section The output dimension equals the input dimension. The reconstruction loss uses Huber loss (the threshold parameter can be set to 1): ;

[0183] in, ;

[0184] A gradient reversal layer is inserted between the center discriminator and the feature extractor. The gradient of the classification loss is backpropagated to the feature extractor through the gradient reversal layer, enabling the feature extractor to learn feature representations independent of the center. Specifically:

[0185] The gradient inversion layer (GRL) acts as an identity mapping during forward propagation: During backpropagation, the gradient is multiplied by a negative coefficient. : ,in It is the identity matrix. To counter the strength coefficient, it is gradually increased from 0 to 1 during training (according to...). The increasing curve, (This represents the training progress percentage, ranging from 0 to 1).

[0186] Through the backpropagation operation of GRL, the gradient of the classification loss with respect to the feature extractor parameters is replaced by − : This reverses the optimization objective of the feature extractor from "helping the center discriminator classify correctly" to reducing the classification accuracy of the center discriminator. Driven by this objective, the latent space feature z learned by the feature extractor will eliminate the statistical correlation with the center label and achieve cross-center invariance of the feature.

[0187] With the joint optimization objectives of minimizing reconstruction loss and maximizing classification loss, adversarial training is performed on the feature extractor, center discriminator, and reconstructor, specifically as follows:

[0188] The loss function for the joint optimization objective is: ;

[0189] in, The positive sign indicates minimizing the reconstruction error. The negative sign (achieved through gradient reversal via GRL) indicates maximizing the classification loss (i.e., preventing the center discriminator from classifying accurately).

[0190] Training uses the Adam optimizer with a learning rate of 0.001 and a batch size of 64. The training run consists of 100 epochs or continues until the validation loss converges. In each epoch, the following steps are performed sequentially: forward propagation calculation... , , Backpropagation calculates the gradient, where The gradient is inverted by GRL and then fed back to the feature extractor;

[0191] Update network parameters, and discard the center discriminator after training is complete. Only retain the feature extractor and refactor For domain alignment residual calculation during the inference phase, for new centers (centers not seen during the training phase), the trained feature extractor and reconstructor are directly used for forward calculation during inference, without the need for a center discriminator.

[0192] In this implementation plan, the natural variation patterns of indicators are captured by combining the time-series characteristics of visits with attention mechanisms, reasonable numerical ranges are predicted and the degree of time-series deviation is quantified, so as to accurately identify cross-visit latent trend anomalies. At the same time, it is adapted to the characteristics of multi-center scientific research data, and the kernel density estimation and Bayesian methods are used to adapt to different center data distributions. The adversarial domain alignment network is used to remove the feature interference brought by the center, and the two-dimensional residuals are fused to jointly determine data anomalies, thereby effectively distinguishing between normal center differences and real data deviations.

[0193] Specifically, the steps for generating topology reconstruction residuals are as follows:

[0194] Based on the hierarchical relationships among visit layers, form layers, and field layers in the hierarchical data (i.e., a patient has multiple visits, each visit contains multiple forms, and each form contains multiple fields), and combined with the constraint edges between field nodes in the dependency graph, a heterogeneous graph is generated. The connection relationships in the heterogeneous graph are determined by the constraint edges and the hierarchical relationships, specifically as follows:

[0195] Heterogeneous graphs are constructed independently for each patient. , where the set of nodes It contains three types of nodes: visit nodes (one node for each actual completed visit), form nodes (one node for each form filled out under each visit), and field nodes (one node for each field in each form). The total number of nodes is usually 20-200, and the edge set is... It contains two types of edges:

[0196] Subordinate relationship edge (hierarchical structure edge): from the patient node to the visit node, from the visit node to the form node, from the form node to the field node, indicating the hierarchical belonging of the data structure;

[0197] Dependency edges: Directly reuse the constraint edges between field nodes in the dependency graph (time order dependency, field logical dependency, consistency dependency), but the edge weight is adjusted as follows: if the dependency is across forms or across visit types, the edge weight is multiplied by 1.2;

[0198] In heterogeneous graphs, nodes at different levels are connected by subordinate edges, and field nodes at the same or different levels are connected by dependency edges, forming a composite graph structure that integrates data structure and logical dependencies.

[0199] Following the connectivity of the graph structure, multiple rounds of feature propagation and aggregation are performed through a pre-trained inductive graph neural network to generate hidden layer features, specifically as follows:

[0200] The inductive graph neural network GraphSAGE is used as the encoder. All patients share the same set of model parameters (the GraphSAGE model is trained in a self-supervised manner, the training data is the heterogeneous graphs of all patients in Dclean, the training objective is to reconstruct the target field value based on neighborhood features, the loss function is Huber loss, and mini-batch stochastic gradient descent is performed on the patient graph with a batch size of 32 and a learning rate of 0.001. Training is stopped after 200 rounds or when the validation loss does not decrease for 10 consecutive rounds). It supports the addition of new patients without rebuilding the entire graph.

[0201] For each field node Its initial eigenvector It consists of the following components: field value (standardized numerical value), field type code (numerical / categorical / date type), field importance level, whether it is a key field, and the feature dimension can be set to 128;

[0202] GraphSAGE performs D rounds of neighbor aggregation. The aggregation formula for the d-th round (d=1, 2, ..., D) is: ; ;

[0203] in, For nodes The set of neighboring nodes (including all nodes connected by dependency edges), aggregate function Mean aggregation is used (averaging the features of neighboring nodes). For the first Layer-learnable weight matrix, It is the ReLU activation function. For vector concatenation operations, after D rounds of aggregation, the final hidden features of each field node are: ;

[0204] Based on hidden layer features, Monte Carlo random inactivation is enabled in the inductive graph neural network for multiple forward propagations to obtain multiple reconstructed values. Bias normalization is then performed to generate topological reconstruction residuals, specifically:

[0205] A Dropout layer is introduced after each layer of the trained GraphSAGE model, with a dropout rate of 0.2. During the inference phase, the same input is processed... Second forward propagation ( Each forward propagation randomly resets the Dropout mask, resulting in... Different hidden layer features ;

[0206] For each field node The hidden layer features are input into a reconstruction head (a single-layer fully connected network, with an output dimension equal to the dimension of the field value: 1 for numeric types and the number of categories for categorical types), to obtain... One reconstructed predicted value ,calculate The mean of the predictions and standard deviation ;

[0207] Topology Reconstruction Residual Defined as the true value The deviation from the predicted mean relative to the predicted standard deviation is truncated to the nearest multiple. Interval: ;

[0208] in, To prevent division by zero for small constants, for categorical fields, Replace with 1 minus the predicted probability (if the true class is 1). The residual is ).

[0209] It should be noted that the training and inference of all deep learning models in this implementation example can be deployed on a general-purpose GPU server. The training of all models can be divided into training set, validation set and test set in a 7:1.5:1.5 ratio according to patient ID to ensure that the data of the same patient is not distributed across sets. The model selection is based on the performance indicators on the validation set to determine the best one.

[0210] The specific steps for generating a comprehensive quality score are as follows:

[0211] Adaptive weighting is applied to rule violation residuals, time series prediction residuals, cross-center alignment residuals, and topology reconstruction residuals to generate a basic quality risk score, specifically as follows:

[0212] During operation, each detector uses "Patient ID_Visit Node_Form Name_Logical Field Name" as the unique identifier for each data point, and outputs the residual value and triggering reason for that data point.

[0213] Traverse all collected field value records in the hierarchical data and generate an initial label vector for each record: ;

[0214] Each component is defined as follows: If the rule validator provides a value for this data point... ,but If the longitudinal trend predictor gives the following for this data point: ,but If the domain alignment detector gives the data point ,but If the dependency graph detector provides the following for this data point: ,but The number of components that did not trigger an exception remains at 0;

[0215] Based on the tag vector Classify the data points:

[0216] like If a data point is identified as abnormal by at least one detector, it is marked as a "suspicious data point" and enters the subsequent comprehensive scoring process.

[0217] like If no abnormality is detected by any detector, the data point is marked as "normal", and the comprehensive score is skipped directly without generating a question record.

[0218] For data points marked as "suspicious", extract four residual values: (Rules violate residuals) (Time series prediction residuals) (Cross-center aligned residuals) (Topology reconstruction residual). The formula for calculating the basic quality risk score is:

[0219] ;

[0220] in, For adaptive weighting coefficients, satisfying The initial weights are set to... It favors time-series prediction and cross-center detection results, and dynamically adjusts the weights based on field importance level and central features:

[0221] In one implementation scheme, the weight adjustments are shown in Table 1:

[0222] Table 1. Examples of Weight Adjustment

[0223] If the sample size of the research center Or visit completion rate , Reduced to 0.1, other weights are renormalized proportionally.

[0224] Downgraded circuit breaker strategy: If the total number of patients in a research center is less than 5, the center's operations will be suspended. and Computation, only depends If a patient's heterogeneous graph has more than 500 nodes, skip. Its weight Distributed proportionally ;

[0225] And extract the important weights of fields and the completeness coefficient of the evidence chain from the field layer, specifically:

[0226] Field importance weight Read directly from the field importance level stored in the field layer. Predefined values: primary endpoint field (1.0), critical security field (0.9), core inbound and outbound criteria field (0.8), and general auxiliary field (0.5).

[0227] Completeness of the chain of evidence The completeness of the evidence available to reflect the current anomaly is determined based on existing metadata (modification history, source data verification markers), and does not depend on subsequent evidence chain preservation processes. ;

[0228] The source data verification flag indicates that the value of this field has been automatically or manually compared with the original source file (such as a PDF test report or original medical record) and confirmed to be consistent.

[0229] Based on the importance weight of fields and the evidence chain integrity coefficient, the basic quality risk score is corrected to obtain the comprehensive quality score, which is as follows:

[0230] The formula for calculating the overall quality score is: ;

[0231] In one implementation, a primary endpoint field ( The residual is The weight is Calculated This field contains modification history and source data verification flags. The overall quality score is .

[0232] In this implementation plan, a heterogeneous graph is constructed based on hierarchical subordination and logical dependency to achieve deep aggregation of multi-node features. The degree of field topological deviation is quantified by combining random deactivation method, thereby accurately capturing cross-level implicit association anomalies. Furthermore, through adaptive fusion of multiple types of residuals and dynamic weight allocation based on field attributes, combined with a reasonable circuit breaker mechanism to adapt to different scenarios, and combined with the importance of fields and the completeness of evidence to correct the score, the accuracy of data anomaly risk can be accurately quantified, thereby improving the accuracy of scientific research data quality verification.

[0233] Please see Figure 3This invention provides a technical solution: a scientific research data quality verification system based on AI machine learning, comprising: a hierarchical structure construction unit for collecting heterogeneous scientific research data, constructing a hierarchical structure data including a patient layer, a visit layer, a form layer, and a field layer through semantic knowledge base construction and automatic field mounting; a dependency mining unit for performing dependency mining on the hierarchical structure data, constructing a dependency graph containing a set of edges, including dependency constraint edges and logical edges; a hierarchical heterogeneity identification unit for performing anomaly identification based on the dependency graph and hierarchical structure data using a pre-built hierarchical heterogeneity identification model, generating rule violation residuals, time series prediction residuals, cross-center alignment residuals, and topology reconstruction residuals; a comprehensive quality scoring unit for comprehensively processing each residual, generating a comprehensive quality score, and automatically classifying the problem level; and an evidence chain storage unit for encrypting and storing evidence chains for problems that meet the preset level, and generating an audit report.

[0234] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0235] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An AI machine learning-based scientific research data quality checking method, characterized in that, Includes the following steps: Heterogeneous scientific research data is collected, and hierarchical data is constructed through semantic knowledge base construction and automatic field mounting, including patient layer, visit layer, form layer and field layer; Dependency mining is performed on hierarchical data to construct a dependency graph containing a set of edges, including dependency constraint edges and logical edges. Based on dependency graphs and hierarchical data, a pre-built hierarchical heterogeneity identification model is used for anomaly identification, including: The rule validator traverses the constraint edges in the dependency graph, performs violation verification, and generates rule violation residuals. The longitudinal trend predictor, based on dependency graphs and hierarchical data, statistically analyzes the longitudinal sequence of visits, and generates time-series prediction residuals through cross-visit temporal attention encoding and trend extrapolation. The domain alignment detector extracts a consistent set of fields based on dependency graphs and hierarchical data, and calculates reconstruction error and dynamic adaptive threshold correction by learning center-invariant features through adversarial training, thereby generating cross-center alignment residuals. The dependency graph detector constructs a heterogeneous graph based on hierarchical data and dependency graphs, and generates topology reconstruction residuals through graph structure propagation and iterative reconstruction. The residuals are processed comprehensively to generate a comprehensive quality score and the problem level is automatically classified. For issues that meet the preset level, the evidence chain is encrypted and stored, and an audit report is generated. 2.The AI machine learning based scientific research data quality checking method according to claim 1, characterized in that, The specific steps for constructing hierarchical data are as follows: Heterogeneous scientific research data is extracted to a temporary storage area, and data tables are obtained from the temporary storage area and classified according to the preset hierarchical mapping rules. The categorized data tables are organized into an initial structure, and a semantic knowledge base containing standard concepts, field alias mappings, unit conversion rules, and default quality control rules is pre-built. The initial structure is vectorized and semantically matched with standard concepts in the semantic knowledge base using semantic similarity calculation. Based on the semantic matching results, semantic attachment and rule inheritance are performed, and the data is backfilled into the initial structure to build a hierarchical data structure. 3.The AI machine learning based scientific research data quality checking method according to claim 2, characterized in that, The specific steps for constructing a dependency graph are as follows: Extract several fields from the field layer and use them as field nodes; Add dependency constraint edges between field nodes that meet preset logical constraints. Logical constraints include time order dependencies, field logical dependencies, and consistency dependencies between forms. Data cleaning and quality pre-screening are performed on hierarchical data to generate qualified scientific research data; Based on qualified scientific research data, statistical dependency mining and causal structure learning methods are used to generate logical edges for different fields. And assign distribution consistency tags to field nodes that meet the preset cross-center consistency conditions; Constraint edges, logical edges, and each field node and its corresponding distribution are consistently labeled and combined to form a dependency graph.

4. The method for verifying the quality of scientific research data based on AI machine learning according to claim 1, characterized in that, The specific steps for generating rules that violate residuals are as follows: Determine the constraint type and corresponding constraint threshold of the constrained edge; Based on the dependency graph, the set of associated field nodes that have constraint edges with the field nodes is extracted; Determine whether the field node and its corresponding associated field node set satisfy the constraint type and corresponding constraint threshold of the constraint edge; If the conditions are not met, constraint deviation mapping is performed on the field node, the corresponding set of associated field nodes, and the constraint threshold to generate rule violation residuals.

5. The method for verifying the quality of scientific research data based on AI machine learning according to claim 1, characterized in that, The specific steps for generating time series prediction residuals are as follows: Based on the field nodes in the dependency graph and the visit and field layers in the hierarchical data, the target field is determined. For the target field, extract the longitudinal sequence of visits in chronological order. The longitudinal sequence of visits is input into the time series prediction model. The time series prediction model uses cross-visit time series attention encoding to perform weighted aggregation of the longitudinal sequence of visits to obtain the field prediction value and the field prediction confidence interval. Extract the field values ​​of the target field from the hierarchical data, and perform confidence residual mapping processing with the predicted field values ​​and the predicted field confidence intervals to generate time series prediction residuals.

6. The method for verifying the quality of scientific research data based on AI machine learning according to claim 3, characterized in that, The specific steps for generating cross-center aligned residuals are as follows: Based on the field nodes marked as having consistent distribution, extract the consistent field set from the hierarchical data and group them. For the grouping results, historical research data are extracted from the hierarchical data, and a dynamic adaptive threshold is generated through kernel density estimation. An adversarial domain adaptation network is constructed, which includes a feature extractor, a center discriminator, and a reconstructor. The network learns irrelevant feature representations through adversarial training and calculates the reconstruction error to obtain the domain alignment residual. The dynamic threshold residual and the domain alignment residual are jointly corrected, and the result of the joint correction is used as the cross-center alignment residual.

7. The method for verifying the quality of scientific research data based on AI machine learning according to claim 6, characterized in that, The construction steps of the adversarial domain adaptation network are as follows: The grouping results are input into the feature extractor, and the latent space feature vector is output after deep feature encoding. The latent space feature vectors are simultaneously input into the center discriminator and the reconstructor; The center discriminator predicts the center label to which the latent space feature vector belongs and calculates the classification loss; the reconstructor reconstructs the latent space feature vector, outputs the reconstructed feature vector, and calculates the reconstruction loss. A gradient reversal layer is inserted between the central discriminator and the feature extractor. The gradient of the classification loss is backpropagated to the feature extractor through the gradient reversal layer, so that the feature extractor learns irrelevant feature representations. The feature extractor, center discriminator, and reconstructor are trained adversarially with the joint optimization objective of minimizing reconstruction loss and maximizing classification loss.

8. The method for verifying the quality of scientific research data based on AI machine learning according to claim 1, characterized in that, The specific steps for generating topology reconstruction residuals are as follows: Based on hierarchical data and combined with constraint edges between field nodes in the dependency graph, a heterogeneous graph is generated, where the connection relationships of the heterogeneous graph are determined by the constraint edges. Along the connectivity of the graph structure, multiple rounds of feature propagation and aggregation are performed through a pre-trained inductive graph neural network to generate hidden layer features; Based on the hidden layer features, Monte Carlo random deactivation is enabled in the inductive graph neural network for multiple forward propagations to obtain multiple reconstruction values. The bias is then normalized to generate topological reconstruction residuals.

9. The method for verifying the quality of scientific research data based on AI machine learning according to claim 1, characterized in that, The specific steps for generating a comprehensive quality score are as follows: Adaptive weighting is applied to rule violation residuals, time series prediction residuals, cross-center alignment residuals, and topology reconstruction residuals to generate a basic quality risk score. And extract the important weights of fields and the integrity coefficient of the evidence chain from the field layer; Based on the importance weight of fields and the evidence chain integrity coefficient, the basic quality risk score is corrected to obtain the comprehensive quality score.

10. A scientific research data quality verification system based on AI machine learning, employing the scientific research data quality verification method based on AI machine learning as described in any one of claims 1-9, characterized in that, include: Hierarchical structure building unit is used to collect heterogeneous scientific research data. After semantic knowledge base construction and automatic field mounting, hierarchical structure data is constructed, which includes patient layer, visit layer, form layer and field layer. The dependency mining unit is used to mine dependencies in hierarchical data and construct a dependency graph containing a set of edges, including dependency constraint edges and logical edges. The hierarchical heterogeneity identification unit is used to identify anomalies based on dependency graphs and hierarchical structure data using a pre-built hierarchical heterogeneity identification model, and to generate rule violation residuals, time series prediction residuals, cross-center alignment residuals, and topology reconstruction residuals. The comprehensive quality scoring unit is used to comprehensively process each residual, generate a comprehensive quality score, and automatically classify the problem level; The evidence chain storage unit is used to encrypt and store evidence chains for issues that meet preset levels and generate audit reports.

Images