Data quality control and automatic repair method and system

Abstract

The application is suitable for the technical field of medical big data quality management, and provides a data quality control and automatic repair method and system. The method comprises the following steps: performing federated desensitization slicing on multi-source medical data; performing double time sequence anchoring to obtain a double time sequence offset; constructing a time sequence-entity double-dimensional dynamic constraint graph based on the offset, and calculating an abnormal confidence to detect abnormalities; performing distributed causal consistency verification on abnormal data, and then performing minimum cost automatic repair; writing the repaired data back to the source system, and performing online incremental update on the constraint graph based on the repaired data. Through the closed loop of detection-verification-repair-write-update, the application realizes dynamic self-adaptive optimization of rules, effectively avoids secondary abnormalities introduced by repair, and improves data quality while ensuring data privacy and time sequence accuracy.

Description

Technical Field

[0001] This application belongs to the field of medical big data quality management technology, and in particular relates to a data quality control and automatic repair method and system. Background Technology

[0002] With the development of medical informatization, hospital information systems, wearable devices, gene sequencing, and other technologies have generated massive amounts of heterogeneous medical data from multiple sources. However, the quality of this data varies greatly, seriously affecting its application value in clinical research, decision support, and other fields. Existing data quality control methods generally face numerous challenges.

[0003] First, the diverse data sources and inconsistent clocks across different acquisition devices lead to severe time-series drift and data out-of-order issues, making accurate time-based correlation analysis difficult. Second, quality control rules are typically statically configured, failing to dynamically adapt to changes in data distribution, resulting in low detection accuracy and recall. Third, data repair processes often lack a global perspective and causal considerations, easily introducing new logical conflicts and secondary anomalies when fixing one error. Furthermore, many systems fail to write high-quality data back to the source system after data repair, resulting in opaque quality control processes, untraceable results, and an inability to continuously optimize rules based on repair effectiveness. More seriously, rule updates often require system downtime for maintenance, failing to meet the high availability requirements of medical operations and posing privacy and security risks related to the leakage of original patient information during processing. Summary of the Invention

[0004] This application provides a data quality control and automatic repair method and system, which aims to solve the technical problems existing in the prior art, such as severe time drift, static and rigid rules, easy secondary conflicts during repair, inability to close the loop in the process, need for downtime for updates, and risk of data leakage.

[0005] In a first aspect, embodiments of this application provide a data quality control and automatic repair method, the method comprising: Acquire multi-source medical data from original data sources; The multi-source medical data is subjected to federated desensitization slicing to obtain federated desensitized data blocks; Perform dual-time anchoring on the federated de-identified data block to obtain the device native timestamp, global logical clock timestamp, and dual-time offset; Based on the dual temporal offsets, a temporal-entity dual-dimensional dynamic constraint map is constructed, and anomaly confidence is calculated based on the temporal-entity dual-dimensional dynamic constraint map to obtain anomaly detection results; Perform distributed causal consistency verification on the data corresponding to the anomaly detection results to determine the abnormal data to be repaired; Perform minimum-cost automatic repair on the abnormal data to be repaired to obtain the repaired data; The repaired data is written back to the original data source, and an online incremental update is performed on the time-series-entity dual-dimensional dynamic constraint map based on the repaired data.

[0006] In one possible implementation of the first aspect, performing federated desensitization slicing on the multi-source medical data to obtain federated desensitized data blocks includes: Identify the data subject identifiers contained in the multi-source medical data; Perform a one-way hash on the data subject identifier to generate an anonymous identifier that cannot be reversed; The multi-source medical data, after undergoing one-way hashing, is fragmented and isolated according to a fixed time window and data domain to obtain the federated de-identified data block.

[0007] In one possible implementation of the first aspect, the global logical clock timestamp is a uniformly allocated, globally unique, and monotonically increasing timestamp. The dual timing offset is the difference between the global logical clock timestamp and the device's native timestamp.

[0008] In one possible implementation of the first aspect, the time-series-entity two-dimensional dynamic constraint graph includes: Use at least one of the following as an entity node: data field, data type, and acquisition device; At least one of the following constraints—format constraint, value range constraint, spatiotemporal consistency constraint, and causal association constraint—is used as a weighted directed constraint edge connecting the entity nodes.

[0009] In one possible implementation of the first aspect, the calculation of anomaly confidence based on the time-entity dual-dimensional dynamic constraint graph to obtain anomaly detection results includes: Calculate the constraint deviation of a single data entry in the federated de-identified data block relative to the directed constraint edge; Calculate the variance of the weights of the directed constraint edges; Calculate the information entropy of the single data entry; By integrating the constraint deviation, the weight of the directed constraint edge, the bi-temporal offset, the weight variance, and the information entropy, an anomaly confidence score that identifies the degree of anomaly of a single data point is calculated. The anomaly detection result is obtained by comparing the anomaly confidence score with a preset threshold.

[0010] In one possible implementation of the first aspect, the calculation of an anomaly confidence score identifying the degree of anomaly of a single data point by fusing the constraint deviation, the weight of the directed constraint edge, the bi-temporal offset, the weight variance, and the information entropy includes: The constraint deviation of the single data point relative to each directed constraint edge is multiplied by the dual time-series offset, and then weighted and aggregated based on the weight of the directed constraint edge to obtain the aggregated deviation. Based on the weight variance and the information entropy, a measure of data uncertainty is determined; Based on the aggregation deviation and the data uncertainty measure, an anomaly confidence score is obtained to identify the degree of anomaly of the individual data.

[0011] In one possible implementation of the first aspect, the distributed causal consistency check includes at least one of the following: Time sequence reversal conflict verification, cross-device causal order reversal conflict verification, and system state consistency verification.

[0012] In one possible implementation of the first aspect, performing minimum-cost automatic repair on the abnormal data to be repaired to obtain repaired data includes: With the goal of minimizing the repair cost and the deviation of the abnormal confidence score before and after repair, the optimal repair vector is determined under the condition of satisfying the constraints of the time-series-entity dual-dimensional dynamic constraint map; the deviation of the abnormal confidence score before and after repair is determined based on the absolute value of the difference between the abnormal confidence score before repair and the abnormal confidence score after repair. The abnormal data to be repaired is repaired based on the optimal repair vector to obtain the repaired data.

[0013] In one possible implementation of the first aspect, the online incremental update of the time-series-entity dual-dimensional dynamic constraint map based on the repaired data includes: Calculate the decrease in anomaly confidence score between the repaired data and the anomaly data to be repaired; wherein the decrease in anomaly confidence score is the difference between the anomaly confidence score before repair and the anomaly confidence score after repair. Based on the decrease in the abnormal confidence score, calculate the weight increment of each directed constraint edge in the time-series-entity dual-dimensional dynamic constraint graph; Based on the current weight and corresponding weight increment of each directed constraint edge, the updated weight of each directed constraint edge is obtained.

[0014] Secondly, embodiments of this application provide a data quality control and automatic repair system, including: The acquisition module is used to acquire multi-source medical data from the original data source; The federated desensitization slicing module is used to perform federated desensitization slicing on the multi-source medical data to obtain federated desensitized data blocks; The dual-time anchoring module is used to perform dual-time anchoring on the federated desensitized data block to obtain the device native timestamp, global logical clock timestamp, and dual-time offset. The graph construction and detection module is used to construct a time-series-entity dual-dimensional dynamic constraint graph based on the dual time-series offsets, and to perform anomaly confidence calculation based on the time-series-entity dual-dimensional dynamic constraint graph to obtain anomaly detection results. The causality verification module is used to perform distributed causal consistency verification on the data corresponding to the anomaly detection results in order to determine the abnormal data to be repaired. The repair module is used to perform automatic repair on the abnormal data to be repaired at the lowest cost, and obtain the repaired data; The update module is used to write the repaired data back to the original data source and perform online incremental updates on the time-series-entity dual-dimensional dynamic constraint map based on the repaired data.

[0015] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data quality control and automatic repair method described in any one of the first aspects above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data quality control and automatic repair method described in any one of the first aspects.

[0017] Fifthly, embodiments of this application provide a computer program product that, when run on a computer device, causes the computer device to execute the data quality control and automatic repair method described in any one of the first aspects.

[0018] This application's embodiments, through federated desensitization slicing, anonymize the data subject identifiers without aggregating the original data, and standardize the data fragmentation. This ensures patient privacy and security while laying the foundation for subsequent unified processing of multi-source heterogeneous data, resolving the risks of data leakage and the problem of multi-source heterogeneity. By introducing a global logical clock and calculating the dual-time-series offset from the device's native timestamp, clock drift between different devices can be accurately quantified and calibrated, solving the difficulties in data correlation analysis and anomaly misjudgment caused by time-series errors, ensuring the accuracy of time-series data. By constructing a time-series-entity dual-dimensional dynamic constraint graph and combining it with an online incremental update mechanism, quality control rules can adaptively adjust based on the actual data repair effect. This system replaces the traditional static rule engine, enabling dynamic rule optimization without system downtime, thus improving system availability and detection accuracy. Before repair, distributed causal consistency verification is performed, pre-identifying and avoiding repair operations that might cause logical conflicts. Minimum-cost automatic repair, while satisfying global constraints, aims to minimize data changes, avoiding secondary anomalies and preserving data authenticity to the greatest extent. By writing back the repaired high-quality data to the original data source and updating the constraint graph online, a complete technical closed loop of detection-verification-repair-writeback-update is formed, making the entire data quality improvement process transparent, traceable, and capable of continuous self-optimization.

[0019] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a data quality control and automatic repair method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a data quality control and automatic repair system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0023] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] Please see Figure 1This application provides a data quality control and automatic repair method, aiming to solve problems in existing technologies such as severe time-series drift in multi-source medical data, static and rigid quality control rules, easy secondary conflicts during the repair process, inability to close the loop, and the risk of data leakage. This method systematically improves the quality and usability of medical data by constructing a complete closed loop from data acquisition to repair and write-back.

[0029] S101, acquires multi-source medical data from the original data source.

[0030] The original data sources include at least one of the following: hospital information systems, wearable devices, gene sequencing platforms, and laboratory information systems.

[0031] Multi-source medical data includes at least one of the following: image metadata, biochemical index data, wearable time-series data, and gene format data.

[0032] In this embodiment, multi-source medical data is obtained from various raw data sources, such as hospital information systems and wearable devices. These data vary in format and quality, and using them directly would pose significant obstacles to subsequent analysis and application. Therefore, obtaining this raw data provides initial input for subsequent unified processing and quality control procedures.

[0033] S102, perform federated desensitization slicing on the above multi-source medical data to obtain federated desensitized data blocks.

[0034] Among them, federalized desensitization slicing refers to a preprocessing method that performs subject identification desensitization and slicing on multi-source data without aggregating the original medical data, in order to achieve data privacy isolation and standardization.

[0035] Among them, the federated de-identified data block refers to a data unit with a unified structure and controllable size formed after de-identification and fragmentation.

[0036] In this embodiment of the application, this step desensitizes and standardizes the data source locally, avoiding the centralized aggregation of raw sensitive data. At the same time, it transforms heterogeneous data into data blocks with a unified structure, providing a secure and standardized data foundation for subsequent distributed computing and analysis steps. This solves the privacy risks and low processing efficiency problems caused by directly processing raw data, and achieves standardization and security in data preprocessing.

[0037] S103, perform dual timing anchoring on the above-mentioned federated de-identified data block to obtain the device's native timestamp, global logical clock timestamp, and dual timing offset.

[0038] Dual time-series anchoring refers to the process of simultaneously assigning a timestamp generated by the device itself and a timestamp uniformly allocated by the system to the same data record in order to establish a time calibration benchmark.

[0039] Among them, the device native timestamp T1 is a timestamp generated locally by the acquisition device; the global logical clock timestamp T2 is a globally unique and monotonically increasing timestamp allocated by the unified closed-loop architecture.

[0040] The dual timing offset ΔT is the difference between the global logical clock timestamp and the device's native timestamp.

[0041] In this embodiment, this step assigns both the device's own timestamp and a globally allocated logical clock to each piece of data, and calculates the offset between the two. This offset quantifies the degree of clock drift for each device. Thus, a globally unified time reference is introduced, resolving the problem of incomparable timestamps from multiple data sources and achieving alignment and reliable sorting of all data in the time dimension.

[0042] S104. Based on the above dual temporal offsets, construct a temporal-entity dual-dimensional dynamic constraint map, and perform anomaly confidence calculation based on the above temporal-entity dual-dimensional dynamic constraint map to obtain anomaly detection results.

[0043] Among them, the time-series-entity dual-dimensional dynamic constraint graph is a graph structure data model with data attributes as nodes and constraint relationships between data as weighted directed edges, used to uniformly describe and manage data quality rules.

[0044] Among them, the calculation of anomaly confidence refers to the process of quantifying the degree of anomaly of a single data point by integrating constraint deviation, constraint edge weight, dual time series offset, weight variance and data information entropy.

[0045] Among them, the anomaly detection result refers to the judgment output that marks the data as normal or abnormal based on the comparison between the anomaly confidence score and the preset threshold.

[0046] In this embodiment, a dynamic graph that integrates data entity constraints (such as value range and format) and temporal constraints (based on dual temporal offsets) is constructed, comprehensively describing the complex rules that data should satisfy. Anomaly detection based on this graph, compared to simple rule matching, can discover deeper time-related anomalies. This approach solves the problems of traditional static rules being unable to adapt to changes in data distribution and having a single detection dimension, achieving dynamic, multi-dimensional, and high-precision detection of data quality.

[0047] S105, Perform distributed causal consistency verification on the data corresponding to the above anomaly detection results to determine the abnormal data to be repaired.

[0048] Distributed causal consistency verification refers to the process of performing time-series reversal, cross-device causal order, and system state consistency verification on abnormal data based on the global logical clock timestamp and causal constraint edges in the constraint graph.

[0049] Among them, abnormal data to be repaired refers to abnormal data that has passed the verification and meets the conditions for repair.

[0050] In this embodiment, there is a strong causal relationship between medical data (e.g., diagnosis precedes treatment), and directly fixing an outlier may disrupt this relationship. Therefore, performing causal consistency verification before fixing ensures that the operation to be fixed will not logically conflict with existing, correct data, guaranteeing the logical correctness and usability of the fixed data.

[0051] S106, perform minimum-cost automatic repair on the above-mentioned abnormal data to be repaired, and obtain the repaired data.

[0052] Among them, minimum cost automatic repair refers to the process of automatically correcting abnormal data by solving for the optimal repair vector, with the goal of minimizing the repair cost and the deviation of the abnormal confidence score before and after repair, while satisfying all constraints of the time-series-entity dual-dimensional dynamic constraint map.

[0053] The repaired data refers to the data obtained through the above repair operations that meets the requirements of the constraint graph and has an anomaly confidence score lower than the preset threshold.

[0054] In this embodiment, an optimal repair scheme can be calculated by establishing an optimization problem aimed at minimizing the repair cost (such as the number of fields modified) and minimizing the changes in data characteristics before and after repair. This approach not only corrects errors but also preserves the original information of the data to the greatest extent possible. This step solves the problems of high cost and data distortion inherent in traditional repair methods, achieving accurate and low-loss repair of abnormal data.

[0055] S107, the repaired data is written back to the original data source, and online incremental updates are performed on the time-series-entity dual-dimensional dynamic constraint map based on the repaired data.

[0056] Write-back refers to overwriting the corrected data with only the corresponding abnormal data in the original data source, thus achieving a closed-loop improvement in data quality at the source.

[0057] Online incremental updates refer to dynamically adjusting the weights of directed constraint edges in the constraint graph without interrupting system operation, based on the repaired data and related statistical information (including the decrease in abnormal confidence scores, repair accuracy, and the pass rate of distributed causal consistency verification).

[0058] In this embodiment, the repaired data is written back to the original data source, ensuring that the data quality improvement is reflected at the source. Furthermore, each repaired data point is used as a learning sample to dynamically adjust the rule weights in the constraint graph, allowing the rule base to adapt to changes in data distribution. This forms a closed-loop feedback mechanism, enabling continuous learning and optimization of the data quality control system.

[0059] In an optional embodiment, the process of performing federated desensitization and slicing of multi-source medical data specifically includes: Step a1: Identify the data subject identifiers contained in the above multi-source medical data.

[0060] Among them, data subject identifiers refer to information that can directly or indirectly identify a patient's personal identity, including at least one of name, medical record number, and mobile phone number.

[0061] Step a2: Perform one-way hashing on the above data subject identifier to generate an anonymous identifier that cannot be reversed.

[0062] One-way hashing refers to the process of using a cryptographic hash function (such as SHA-256) to convert an input of arbitrary length into a hash value of fixed length, and the original input cannot be recovered from the hash value.

[0063] Anonymous identifiers refer to identifiers generated after one-way hashing that cannot be directly or indirectly associated with the identity of the original data subject.

[0064] Step a3: After performing one-way hashing, the above multi-source medical data is fragmented and isolated according to a fixed time window and data domain to obtain the above federated de-identified data blocks.

[0065] Among them, a fixed time window refers to dividing the time interval of data according to a preset time length (e.g., 5 minutes).

[0066] The data domain refers to the business category of medical data, including at least one of imaging, biochemistry, vital signs, and genes.

[0067] Among them, fragmentation and isolation refers to splitting data into independent, non-overlapping data blocks according to time windows and data domains, and storing or processing them separately.

[0068] In this embodiment, sensitive information such as patient IDs is processed using one-way hashing (such as the SHA-256 algorithm), ensuring that even if data blocks are leaked, the individual's true identity cannot be deduced, thus solving the technical challenge of privacy protection. Simultaneously, data is fragmented according to fixed time windows (adjustable based on actual business needs, 5 minutes / window is optional), dividing heterogeneous data streams into standardized, manageable data units. This lays the foundation for subsequent distributed processing and time-series analysis, achieving standardization of data preprocessing.

[0069] In an optional embodiment, the global logical clock timestamp is a uniformly allocated, globally unique, and monotonically increasing timestamp; the dual timing offset is the difference between the global logical clock timestamp and the device's native timestamp.

[0070] Among them, the original timestamp of the above-mentioned device is generated locally by the acquisition device and may drift, with the drift range typically being ±1000 milliseconds; the above-mentioned global logical clock timestamp is a globally unique and monotonically increasing timestamp that is uniformly allocated; the above-mentioned dual timing offset is the difference between the above-mentioned global logical clock timestamp and the above-mentioned original timestamp of the device, and its value range is from 0 milliseconds to 1000 milliseconds.

[0071] In this embodiment, the global logical clock timestamp serves as the standard time. Its globally unique and monotonically increasing characteristics are fundamental to ensuring the time-series comparability of all cross-device and cross-source data. The dual timing offset precisely quantifies the inaccuracy of each device's local clock. This approach solves the timing chaos problem caused by relying solely on the device's native timestamp, providing an accurate time reference for subsequent time-series-dependent operations (such as causality checks and timing constraint detection).

[0072] In an optional embodiment, the above-mentioned time-series-entity dual-dimensional dynamic constraint graph includes at least one of data fields, data types, and acquisition devices as entity nodes; and at least one of format constraints, value range constraints, spatiotemporal consistency constraints, and causal association constraints as weighted directed constraint edges connecting the above-mentioned entity nodes.

[0073] Each directed constraint edge is assigned a weight to indicate the importance of the constraint. The weight ranges from 0.5 to 3.0, with the initial value preset according to the constraint type (1.0 for format constraints, 1.2 for range constraints, 1.5 for spatiotemporal constraints, and 2.0 for causal constraints). The weight can be dynamically updated based on the repair results. The higher the weight, the greater the impact of the constraint on the validity of the data.

[0074] In this embodiment, the above scheme specifically defines the composition of the constraint graph. Data attributes (fields, types, devices) are abstracted as nodes in the graph, and the rules (format, range, spatiotemporal, causal) that they must satisfy are abstracted as weighted edges. This graph structure can express complex relationships between data more flexibly and comprehensively than traditional rule lists. In this way, the problem of traditional rule engines struggling to describe and manage complex constraints across fields and tables is solved, providing a data model foundation for unified and dynamic quality inspection and repair.

[0075] In an optional embodiment, the process of calculating anomaly confidence based on the aforementioned time-series-entity dual-dimensional dynamic constraint graph to obtain anomaly detection results specifically includes: Step b1: Calculate the constraint deviation of a single data entry in the above-mentioned federated desensitized data block relative to the above-mentioned directed constraint edge.

[0076] In this context, a single data entry in the aforementioned federated desensitized data block refers to an independent record within a structured data unit obtained after desensitization slicing, corresponding to a medical test, a device acquisition, or a log event.

[0077] Among them, constraint deviation refers to the degree of difference between the data and the constraint edge requirements. The value range is from 0 to 1, where 0 means that the constraint is fully met and 1 means that the constraint is not met at all.

[0078] Step b2: Calculate the weight variance of the above directed constraint edges.

[0079] Among them, the weight variance refers to the variance of the weights of all directed constraint edges, which is used to measure the dispersion of the weights of each constraint edge.

[0080] Step b3: Calculate the information entropy of the above single data entry.

[0081] Information entropy is a metric used to measure the uncertainty of the values ​​of each field in a single data entry.

[0082] Step b4: By integrating the above constraint deviation, the weight of the above directed constraint edge, the above bi-temporal offset, the above weight variance, and the above information entropy, an anomaly confidence score that identifies the degree of anomaly of the above single data point is calculated.

[0083] The anomaly confidence score is a non-negative real number. The larger the value, the higher the degree of data anomaly. The anomaly confidence score of normal data is lower than a preset threshold (e.g., 2.0), while the anomaly confidence score of abnormal data is higher than this threshold.

[0084] Step b5: Based on the comparison between the above anomaly confidence score and the preset threshold, the above anomaly detection result is obtained.

[0085] The preset threshold is a critical value used to distinguish between normal and abnormal data. Its value ranges from 1.5 to 2.5, and the default value is 2.0.

[0086] Among them, the anomaly detection result refers to the classification output that marks a single data point as normal or abnormal.

[0087] In this embodiment, the above scheme illustrates the algorithmic logic of anomaly detection, which is not a simple rule violation count, but a comprehensive evaluation across multiple dimensions. Constraint deviation measures the degree to which data violates the rules, while weight variance and information entropy serve as calibration factors, respectively measuring the uncertainty of the rules themselves and the inherent disorder of the data. Dual temporal offsets also take into account anomalies in the time dimension. By integrating these dimensions, the problem of false positives and false negatives easily generated by single-rule judgment is solved, achieving a more accurate quantification of the degree of data anomalies.

[0088] Furthermore, the process of obtaining the anomaly confidence score through the above fusion calculation can specifically include: Step c1: Multiply the constraint deviation of the above single data relative to each directed constraint edge by the above dual time-series offset, and then perform weighted aggregation based on the weight of the above directed constraint edge to obtain the aggregated deviation.

[0089] In this embodiment, the constraint deviation of the data relative to the current constraint edge, the weight corresponding to the constraint edge, and the two-time-series offset are obtained. Then, the constraint deviation is multiplied by the two-time-series offset, and then multiplied by the weight of the constraint edge to obtain the contribution value of the constraint edge to the degree of data anomaly. Finally, the contribution values ​​of all constraint edges are summed to obtain the aggregated deviation. This aggregated deviation reflects the comprehensive deviation of the data across all constraint dimensions and is the aggregated term in the subsequent calculation of the anomaly confidence score.

[0090] Step c2: Based on the aforementioned weight variance and information entropy, determine the data uncertainty measure.

[0091] Among them, the data uncertainty measure is the square root of the sum of the weight variance and the information entropy, which is used to normalize the aggregation deviation to eliminate the bias caused by the weight dispersion and the disorder of the data itself in the anomaly judgment.

[0092] Step c3: Based on the above aggregation deviation and the above data uncertainty measure, obtain the anomaly confidence score that identifies the degree of anomaly of the above individual data.

[0093] In this embodiment, the aggregation deviation is divided by the data uncertainty measure to obtain the anomaly confidence score, which identifies the degree of anomaly of a single data point. That is, the anomaly confidence score = aggregation deviation / data uncertainty measure. This score is a non-negative real number; a larger value indicates a higher degree of anomaly.

[0094] In this embodiment, the design is a specific algorithmic implementation of the aforementioned fusion calculation process. It clarifies the calculation methods for the two intermediate variables, aggregation deviation and data uncertainty measurement, making the entire anomaly confidence score calculation process clearer and more feasible. Through this structured calculation, multiple influencing factors are logically combined, solving the technical problem of how to effectively fuse multi-dimensional information for anomaly detection and enhancing the operability of the solution.

[0095] In an optional embodiment, the above-mentioned distributed causal consistency verification includes at least one of time-reversal conflict verification, cross-device causal order reversal conflict verification, and system state consistency verification.

[0096] This application's embodiments list specific verification types. Time-reversal conflict verification (e.g., checking if the discharge date is earlier than the admission date) and cross-device causal order reversal conflict verification (e.g., checking if medication records are earlier than diagnosis records) are both aimed at common logical errors in medical data. System state consistency verification ensures that repair operations will not contradict data in other parts of the system. By performing these specific verifications, the problem of blindly repairing data potentially introducing secondary logical errors is solved, improving the usability and reliability of the repaired data.

[0097] Furthermore, the aforementioned process of performing minimum-cost automatic repair on the abnormal data to be repaired specifically includes: Step d1 aims to minimize the repair cost and the deviation of the abnormal confidence score before and after repair. Under the constraint of the above time-series-entity dual-dimensional dynamic constraint map, the optimal repair vector is determined. The deviation of the abnormal confidence score before and after repair is determined based on the absolute value of the difference between the abnormal confidence score before repair and the abnormal confidence score after repair.

[0098] The repair cost is used to measure the cost of the repair operation. It is calculated based on the number of fields modified and / or the number of format conversions. The more fields modified and the more complex the format conversions, the greater the repair cost.

[0099] Among them, the deviation of abnormal confidence scores before and after repair refers to the abnormal confidence score before repair. Compared with the abnormal confidence score after repair The absolute value of the difference, i.e. This is used to measure the degree to which the repair operation changes the features of the original data; the smaller the deviation, the closer the repaired data is to the original data. The optimal repair vector refers to the repair scheme that minimizes the objective function (see below) among all feasible candidate repair vectors.

[0100] Step d2: Based on the above-mentioned optimal repair vector, perform repair on the above-mentioned abnormal data to be repaired to obtain the above-mentioned repaired data.

[0101] In this embodiment, the optimal repair vector includes the following information: the field name to be modified, the repaired value, the repair operation type (including at least one of missing value completion, outlier replacement, format normalization, and temporal alignment), and the repair time. Based on the repair operation type and the repaired value in the optimal repair vector, the corresponding field in the outlier data to be repaired is modified, and the repair operation is performed to obtain the repaired data. After the repair is completed, the outlier confidence score of the data can be optionally recalculated to confirm that it is below a preset threshold, thus ensuring the effectiveness of the repair.

[0102] In this embodiment, the optimization objective and constraints for minimizing cost are clearly defined. The repair process is modeled as a constrained bi-objective optimization problem: both cost-effective repair (small modification amount) and effective repair (data more closely resembling normalcy after repair). By solving this optimization problem, an optimal balance point can be automatically found. This approach addresses the subjectivity and arbitrariness of manual repair, as well as the data distortion that may result from simple rule-based repair, achieving automation and optimization of the repair process.

[0103] In an optional embodiment, the process of performing online incremental updates to the aforementioned time-series-entity dual-dimensional dynamic constraint map based on the repaired data specifically includes: Step e1: Calculate the decrease in the abnormal confidence score between the repaired data and the abnormal data to be repaired; wherein the decrease in the abnormal confidence score is the difference between the abnormal confidence score before repair and the abnormal confidence score after repair.

[0104] The decrease in the abnormal confidence score is defined as the difference between the abnormal confidence score before repair and the abnormal confidence score after repair. This decrease reflects the degree to which the repair operation improves the data quality, and the larger the decrease, the more obvious the repair effect.

[0105] Step e2: Based on the above abnormal confidence score decrease, calculate the weight increment of each directed constraint edge in the above time-series-entity two-dimensional dynamic constraint graph.

[0106] The weight increment refers to the magnitude of the adjustment to the weight of each directed constraint edge, reflecting the contribution of that constraint edge to the current repair process. It is calculated as follows: for all records participating in this repair, first calculate the impact of each directed constraint edge on each repaired record, multiply this by the repair accuracy and overall pass rate of that record, then calculate the average, and finally multiply by the attenuation factor.

[0107] Step e3: Based on the current weight and corresponding weight increment of each directed constraint edge, obtain the updated weight of each directed constraint edge.

[0108] The updated weight refers to the final weight value obtained by adding the current weight to the weight increment and correcting for out-of-bounds errors. It is used for quality control of the next batch of data.

[0109] This application describes the system optimization mechanism in detail in its embodiments. The decrease in the anomaly confidence score is used to quantify the degree of improvement in data quality brought about by the repair operation. This decrease is allocated to the directed constraint edges that play a role in the repair process, resulting in the weight increment of each directed constraint edge. Thus, the weights of the constraint edges are updated based on feedback from the repair practice, replacing the method of manually maintaining the rule base, and allowing the rule weights to adapt to changes in data distribution.

[0110] The specific embodiments of this application will be described in more detail below. The overall method flow of this application can be found in [reference needed]. Figure 1 It clearly demonstrates that starting from the acquisition of multi-source medical data from the original data source (S101), it goes through seven steps in sequence, including federated desensitized slicing (S102), dual time-series anchoring (S103), map construction and anomaly detection (S104), distributed causal consistency verification (S105), minimum cost automatic repair (S106), and finally closed-loop write-back and online update (S107), forming a complete closed loop.

[0111] In one specific embodiment, multi-source medical data is first obtained from the original data source (S101).

[0112] Next, federated de-identification slicing is performed (S102). This step aims to preprocess the received multi-source medical data to achieve data privacy protection and standardization. Specifically, the system accesses image metadata from different departments, biochemical indicator data from laboratories, and time-series data from wearable devices. For data subject identifiers such as patient names and ID numbers contained in these data, a one-way hash algorithm such as SHA-256 is used to generate irreversibly anonymized identifiers, ensuring that the original identity information is not leaked in subsequent processes. Subsequently, these de-identified data are segmented and isolated according to preset fixed time windows (e.g., every 5 minutes) and data business domains (e.g., images, biochemistry, vital signs), forming uniformly structured and controllable-size federated de-identified data blocks. Each data block is assigned a globally unique identifier for association and traceability throughout the process. This step provides a secure and standardized data foundation for all subsequent steps.

[0113] Next, dual-time anchoring (S103) is performed. This step targets each data record in the federated, de-identified data block generated in the previous step, aiming to resolve the time asynchrony issue between multiple source devices. Specifically, the system extracts the device-native timestamp T1, generated locally by the acquisition device, from the metadata of each data record. This timestamp may drift by several seconds or even minutes. Simultaneously, the system maintains a high-precision global logical clock, assigning a globally unique and monotonically increasing global logical clock timestamp T2 to this incoming data record. Subsequently, the dual-time offset ΔT = T is calculated. 2- T1. This ΔT value precisely quantifies the degree of clock drift of the data at the acquisition time. It will serve as a key parameter for subsequent timing-related constraint verification and anomaly detection, providing a reliable benchmark for the timing consistency of the entire system.

[0114] Next, the graph construction and anomaly detection are performed (S104). This step, based on the output of the previous step, constructs a unified constraint model and uses it for anomaly identification. First, the system parses the metadata of the federated and de-identified data blocks, extracting data fields (such as blood glucose values), data types (such as numerical types), and acquisition devices (such as a specific model of blood glucose meter) as entity nodes in the time-series-entity dual-dimensional dynamic constraint graph. Then, based on medical knowledge and business rules, weighted directed constraint edges are established between these nodes. For example, a format-oriented directed constraint edge is established between the blood glucose value field and the numerical type, and a value range-oriented directed constraint edge is established between the blood glucose value field and its reasonable range (3.9-6.1 mmol / L). The dual-time-series offset ΔT calculated in the previous step is embedded into the attributes of the spatiotemporal consistency directed constraint edge. The allowable range of ΔT is set to 0-500 milliseconds, and ΔT≥0. Each directed constraint edge is assigned an initial weight w. e For example, format constraints w e =1.0, causal constraint w e =2.0, and its value range is limited to between 0.5 and 3.0. If the updated weight exceeds this range, it will be forcibly corrected to the corresponding boundary value (0.5 for values ​​below 0.5, and 3.0 for values ​​above 3.0). After constructing the graph, for each data point D, its anomaly confidence score is calculated using the following time-series-graph fusion anomaly confidence score formula. : (Formula 1) in, Let be the anomaly confidence score of data D at processing time t; D is a single medical data record, referring to a single data record read from the federated de-identified data block; t is the data processing time, referring to the current system clock timestamp; e is the directed constraint edge number, used to traverse the constructed set of directed constraint edges E; E is the set of directed constraint edges, referring to all directed constraint edges in the time-series-entity constraint graph; we The weight of the e-th directed constraint edge is given by the subscript e, which corresponds to the number of the directed constraint edge. It is used for the calculation of the weight of the directed constraint edge and the value is taken from the initial preset in S104 or the update result in S107. This represents the deviation between data D and the directed constraint edge e at processing time t, used to measure the degree of fit between the data and the constraint. The calculation rules correspond to the directed constraint edge type in S104; specifically, the calculation follows these rules: For format constraints, if the data format meets the requirements... =0, otherwise =1; For range constraints =|Actual value - Reasonable value| / Reasonable interval length. If the result is greater than 1, take 1; if less than 0, take 0. For spatiotemporal constraints... =|ΔT - Maximum Allowable Offset| / Maximum Allowable Offset, where the maximum allowable offset is 500 milliseconds. The result is set to 1 if greater than 1 and 0 if less than 0. For causal constraints, if the causal order meets the requirements... =0, otherwise =1; ΔT is the dual timing offset, representing the difference between the global logical clock timestamp T2 and the device's native timestamp T1; The variance of the directed constraint edge weights is used to measure the dispersion of all directed constraint edge weights in S104. , where is the variance of the weights of the directed constraint edges; , Ent(D) is the arithmetic mean of the weights of all directed constraint edges; Ent(D) is the information entropy of data D, used to measure the degree of disorder in a single data point D. The value range is 0~ M represents the number of possible values ​​for the field to be evaluated in the current data record; p(x) is the probability of each field value in data D, calculated as the number of times a certain field value appears divided by the total number of times all values ​​of that field appear. When the calculated... If the value exceeds a preset threshold (e.g., 2.0), the data is considered abnormal. This threshold can be adjusted within the range of 1.5 to 2.5 according to actual business needs.

[0115] Subsequently, a distributed causal consistency check (S105) is performed. For data identified as anomalous in the previous step, the system does not immediately attempt repair but instead performs this check first to prevent the repair operation from introducing new logical conflicts. The check includes at least: time-series reversal conflict checks within the same subject, such as checking if the sample collection timestamp T2 for the same patient is later than the corresponding test report timestamp T2; cross-device causal order reversal conflict checks, such as checking if the power-on record timestamp T2 of a wearable device is later than the heart rate record timestamp T2; and system state consistency checks to ensure that the data to be repaired is consistent with the relevant data already stored in the system in key information (such as anonymity identifiers and time ranges). If an anomalous data point passes all checks, the next step of the repair process begins. If the process fails, a timing alignment preprocessing will be attempted, which involves reordering the conflicting data according to time based on the global logical clock timestamp T2, and then performing the verification in this step again. If the process still fails, the data will be marked as unrepairable and its abnormal information will be recorded to avoid invalid or harmful repair operations. Such data will not enter the S106 repair process and will not be included in the S107 online incremental update. The system can be configured to periodically clean up the data or trigger a manual review alarm.

[0116] Next, minimum-cost automatic repair (S106) is performed on the verified abnormal data. The goal of this step is to repair the abnormal data at the lowest cost while satisfying all constraints. The system generates multiple candidate repair vectors V for the abnormal data. For example, for a blood glucose value outside the range, candidate solutions may include replacing it with the upper limit of the range, replacing it with the average value of the same batch, etc. Then, each candidate repair vector V is evaluated using the following minimum-cost automatic repair formula, and the optimal repair vector is selected. : (Formula 2) in, λ is the optimal repair vector, representing the repair scheme corresponding to minimizing the objective function; V is the candidate repair vector, referring to the set of all feasible repair schemes; λ is a preset cost balancing coefficient (e.g., 0.5), used to balance the repair cost and the deviation of the abnormal confidence score, with a value range of 0.3 to 0.7; Cost(V) is the repair cost of the candidate repair vector V, used to calculate the cost required for repair, calculated based on the number of modified fields and / or the number of format conversions, such as the number of modified fields × field weight + number of format conversions × 0.2, where the weight of key fields (including anonymous identifier and collection time) is 1.0, and the weight of ordinary fields is 0.5; The outlier confidence score of the repaired data is compared with that in Formula 1. The meanings are the same (both are abnormal confidence scores), only the input parameters are different (V corresponds to the data after repair D', and D corresponds to the data before repair), and the calculation method is the same as S104; To determine the abnormal confidence score of the data D before repair, and in Formula 1... The meanings are consistent, directly calling the calculation result of S104; D' represents the repaired data, referring to the data obtained after repair with minimum cost, which needs to be written back to the original data source after repair. Finally, based on the calculated optimal repair vector... To perform specific repair operations and obtain the repaired data D'.

[0117] It should be noted that, in this embodiment of the application, the anomaly confidence score is written in two different forms depending on the use case: and Both are essentially the same, representing the outlier confidence score of data D. Among them, It is the complete form, explicitly dependent on the processing time t, used for anomaly detection to emphasize the potential impact of the dynamic updating of constraint graph weights over time (online incremental updates) on the score; This is the abbreviated form, omitting time t. It is used for minimum-cost automatic repair because the constraint graph state is fixed during the repair process (the next update has not yet been triggered), and t implicitly remains unchanged. Omitting it makes the formula more concise and unambiguous. Both forms are used interchangeably in the document, with completely consistent meanings and no redundant definitions.

[0118] Finally, closed-loop write-back and online update are performed (S107). This step is the final and self-optimization stage of the entire process. First, the system accurately writes back the verified and effective repaired data D' obtained in the previous step to its original data source, only overwriting the corresponding abnormal data, thereby achieving closed-loop improvement of data quality at the source. Simultaneously, a traceability log containing information from data acquisition to repair completion is generated for subsequent auditing and analysis. More importantly, the system uses the results of this repair practice to perform online incremental updates to the time-series-entity dual-dimensional dynamic constraint graph. This process calculates the weight increment of each directed constraint edge using the following federated distillation weight incremental update formula. : (Formula 3) in, α is the weight increment of the directed constraint edge e, used to represent the update amount of the directed constraint edge weight; α is the federated distillation decay factor, used to control the size of the weight increment, with a value range of 0 to 1, such as 0.8; N is the number of repair records participating in the update, referring to the number of data records successfully repaired in the current batch in S106; i is the repair record sequence number, used to traverse the repair records of the current batch in S106. Let be the influence degree of the directed constraint edge e on the i-th repair record, used to measure the impact of the directed constraint edge on the repair operation. If the optimal repair vector... If the performed repair operation (including at least one of modifying fields, replacing values, or time-series alignment) directly or indirectly changes the matching state between the data and the directed constraint edge e, then... ,otherwise .

[0119] Optional, One example calculation method is as follows: First, calculate the difference between the deviation of the data before repair relative to the directed constraint edge e and the deviation of the data after repair relative to the directed constraint edge e (i.e., the decrease in deviation). Then, divide this decrease by the sum of the decreases in deviation of all constraint edges (adding a small positive number to prevent division by zero). The resulting ratio is the influence of the constraint edge. The larger this value, the more significant the role of the constraint edge in this repair. Another example calculation method is as follows: If the repair operation (such as modifying fields, replacing values, time alignment, etc.) involves data fields associated with the directed constraint edge e, the influence is 1; otherwise, it is 0. The optimal repair vector for the i-th repair record is given by formula 2. The meaning of (optimal repair vector) is consistent, only the index i is added to distinguish different repair records; Let be the repair accuracy of the i-th repaired record, used to measure the repair effect of a single repaired record. Its value ranges from 0 to 1, with a value closer to 1 indicating higher repair quality. Optionally, one example calculation method is to count the number of directed constraint edges that are still not satisfied after repair, and subtract the ratio of this number to the total number of constraint edges involved in the data from 1, i.e., repair accuracy = 1 - (number of constraint edges that are still not satisfied after repair / total number of constraint edges related to the data); another calculation method is: repair accuracy = 1 - (decrease in anomaly confidence score before and after repair / anomaly confidence score before repair), or determined by the correctness of manually labeled repair; where, "still not satisfied" means that after the minimum cost automatic repair operation, the data still cannot satisfy (i.e. violates) the rule defined by a certain directed constraint edge; This is the distributed causal consistency verification pass rate, used to represent the proportion of the current batch that passes verification in S105. The updated weight of the directed constraint edge *e* refers to the weight of the directed constraint edge obtained after updating using Formula 3. This updated weight is used for quality control of the next batch of data. Through the above formula, the weights of the directed constraint edges can be automatically adjusted based on the actual effect of the repair. Updated weights This will be used for quality control of the next batch of data, thus forming a closed loop of continuous learning and optimization. Among these, This represents the original weights of the directed constraint edges. Add weight increment Then, the result is restricted to the interval [0.5, 3.0].

[0120] In an optional embodiment, the above-described anomaly detection steps can be implemented using different technologies. The principle is to leverage the effective learning capabilities of graph neural networks to automatically learn complex patterns of data anomalies from the constructed constraint graph, replacing or supplementing formula-based anomaly confidence calculation methods. In this embodiment, the overall process is consistent with the above embodiment, but after constructing the temporal-entity dual-dimensional dynamic constraint graph, instead of using Formula 1 to calculate the score, the graph is used as input to a pre-trained graph neural network model (such as GCN). This model learns node embeddings and neighborhood information, directly outputting the anomaly probability score for each data node or its associated record. When the score exceeds a preset threshold, it is determined to be abnormal data. Subsequent verification, repair, and update steps remain unchanged, where the online update object can be not only the directed constraint edge weights but also parameters used to fine-tune the graph neural network model. By introducing graph neural networks, more complex anomaly patterns can be captured than those based on fixed formulas, improving detection accuracy and generalization ability.

[0121] In another alternative embodiment, a more rigorous multi-objective optimization strategy can be employed for the automatic repair phase. The principle is to define the repair problem as a multi-objective optimization problem and use a corresponding solution algorithm to avoid the subjectivity and suboptimal solution risks associated with using a single balance coefficient λ. In this embodiment, the repair cost Cost(V) and the abnormal confidence score bias | - These are treated as two independent optimization objectives, solved using multi-objective optimization algorithms such as NSGA-II. This algorithm generates a set of Pareto optimal solutions, i.e., a set of remedial solutions that cannot improve one objective without sacrificing the other. Decision-makers can select the final optimal remedial vector from this set of solutions based on actual business needs (e.g., choosing the lowest-cost solution in cost-sensitive scenarios, or the solution with the smallest deviation in quality-sensitive scenarios). This approach makes the remediation process more flexible and precise.

[0122] The specific values ​​of the coefficients involved in this application can be determined through experimental calibration, offline validation set testing, or online adaptive algorithms. This application does not limit the specific numerical mapping relationship.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0124] Corresponding to the data quality control and automatic repair method described in the above embodiments, Figure 2A structural block diagram of the data quality control and automatic repair system provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0125] Reference Figure 2 The data quality control and automatic repair system includes: The acquisition module is used to acquire multi-source medical data from the original data source; The federated desensitization slicing module is used to perform federated desensitization slicing on the above multi-source medical data to obtain federated desensitized data blocks; The dual-time anchoring module is used to perform dual-time anchoring on the above-mentioned federated de-identified data blocks to obtain the device's native timestamp, global logical clock timestamp, and dual-time offset. The graph construction and detection module is used to construct a time-series-entity dual-dimensional dynamic constraint graph based on the above dual-time-series offsets, and to perform anomaly confidence calculation based on the above time-series-entity dual-dimensional dynamic constraint graph to obtain anomaly detection results. The causality verification module is used to perform distributed causal consistency verification on the data corresponding to the above anomaly detection results in order to determine the abnormal data to be repaired. The repair module is used to perform automatic repair on the above-mentioned abnormal data to be repaired at the lowest cost, and obtain the repaired data; The update module is used to write back the repaired data to the original data source and perform online incremental updates on the time-series-entity dual-dimensional dynamic constraint map based on the repaired data.

[0126] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0128] This application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0129] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0130] This application provides a computer program product that, when run on a computer device, enables the computer device to execute the steps described in the various method embodiments above.

[0131] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 3 As shown, the computer device of this embodiment includes: at least one processor 30 ( Figure 3 (Only one is shown in the diagram), memory 31, and computer program 32 stored in the memory 31 and executable on the at least one processor 30, wherein the processor 30 executes the computer program 32 to implement the steps in any of the above-described data quality control and automatic repair method embodiments.

[0132] The computer device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. They may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, they may also include input / output devices, network access devices, etc.

[0133] The processor 30 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0134] In some embodiments, the memory 31 may be an internal storage unit of the computer device, such as a hard drive or memory. In other embodiments, the memory 31 may be an external storage device of the computer device, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD), flash card, etc. Furthermore, the memory 31 may include both internal and external storage units of the computer device. The memory 31 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / computer equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0136] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data quality control and automatic repair method, characterized in that, The method includes: Acquire multi-source medical data from original data sources; The multi-source medical data is subjected to federated desensitization slicing to obtain federated desensitized data blocks; Perform dual-time anchoring on the federated de-identified data block to obtain the device native timestamp, global logical clock timestamp, and dual-time offset; Based on the dual temporal offsets, a temporal-entity dual-dimensional dynamic constraint map is constructed, and anomaly confidence is calculated based on the temporal-entity dual-dimensional dynamic constraint map to obtain anomaly detection results; Perform distributed causal consistency verification on the data corresponding to the anomaly detection results to determine the abnormal data to be repaired; Perform minimum-cost automatic repair on the abnormal data to be repaired to obtain the repaired data; The repaired data is written back to the original data source, and an online incremental update is performed on the time-series-entity dual-dimensional dynamic constraint map based on the repaired data.

2. The method according to claim 1, characterized in that, The process of performing federated desensitization slicing on the multi-source medical data to obtain federated desensitized data blocks includes: Identify the data subject identifiers contained in the multi-source medical data; Perform a one-way hash on the data subject identifier to generate an anonymous identifier that cannot be reversed; The multi-source medical data, after undergoing one-way hashing, is fragmented and isolated according to a fixed time window and data domain to obtain the federated de-identified data block.

3. The method according to claim 1, characterized in that, The global logical clock timestamp is a uniformly allocated, globally unique, and monotonically increasing timestamp; The dual timing offset is the difference between the global logical clock timestamp and the device's native timestamp.

4. The method according to claim 1, characterized in that, The time-series-entity two-dimensional dynamic constraint graph includes: Use at least one of the following as an entity node: data field, data type, and acquisition device; At least one of the following constraints—format constraint, value range constraint, spatiotemporal consistency constraint, and causal association constraint—is used as a weighted directed constraint edge connecting the entity nodes.

5. The method according to claim 4, characterized in that, The anomaly confidence calculation based on the time-series-entity dual-dimensional dynamic constraint graph yields anomaly detection results, including: Calculate the constraint deviation of a single data entry in the federated de-identified data block relative to the directed constraint edge; Calculate the variance of the weights of the directed constraint edges; Calculate the information entropy of the single data entry; By integrating the constraint deviation, the weight of the directed constraint edge, the bi-temporal offset, the weight variance, and the information entropy, an anomaly confidence score that identifies the degree of anomaly of a single data point is calculated. The anomaly detection result is obtained by comparing the anomaly confidence score with a preset threshold.

6. The method according to claim 5, characterized in that, The method integrates the constraint deviation, the weight of the directed constraint edge, the bi-temporal offset, the weight variance, and the information entropy to calculate an anomaly confidence score that identifies the degree of anomaly of a single data point, including: The constraint deviation of the single data point relative to each directed constraint edge is multiplied by the dual time-series offset, and then weighted and aggregated based on the weight of the directed constraint edge to obtain the aggregated deviation. Based on the weight variance and the information entropy, a measure of data uncertainty is determined; Based on the aggregation deviation and the data uncertainty measure, an anomaly confidence score is obtained to identify the degree of anomaly of the individual data.

7. The method according to claim 5, characterized in that, The distributed causal consistency check includes at least one of the following: Time sequence reversal conflict verification, cross-device causal order reversal conflict verification, and system state consistency verification.

8. The method according to claim 7, characterized in that, The step of performing minimum-cost automatic repair on the abnormal data to be repaired, to obtain repaired data, includes: With the goal of minimizing the repair cost and the deviation of the abnormal confidence score before and after repair, the optimal repair vector is determined under the condition of satisfying the constraints of the time-series-entity dual-dimensional dynamic constraint map; the deviation of the abnormal confidence score before and after repair is determined based on the absolute value of the difference between the abnormal confidence score before repair and the abnormal confidence score after repair. The abnormal data to be repaired is repaired based on the optimal repair vector to obtain the repaired data.

9. The method according to claim 8, characterized in that, The online incremental update of the time-series-entity dual-dimensional dynamic constraint map based on the repaired data includes: Calculate the decrease in anomaly confidence score between the repaired data and the anomaly data to be repaired; wherein the decrease in anomaly confidence score is the difference between the anomaly confidence score before repair and the anomaly confidence score after repair. Based on the decrease in the abnormal confidence score, calculate the weight increment of each directed constraint edge in the time-series-entity dual-dimensional dynamic constraint graph; Based on the current weight and corresponding weight increment of each directed constraint edge, the updated weight of each directed constraint edge is obtained.

10. A data quality control and automatic repair system, characterized in that, include: The acquisition module is used to acquire multi-source medical data from the original data source; The federated desensitization slicing module is used to perform federated desensitization slicing on the multi-source medical data to obtain federated desensitized data blocks; The dual-time anchoring module is used to perform dual-time anchoring on the federated desensitized data block to obtain the device native timestamp, global logical clock timestamp, and dual-time offset. The graph construction and detection module is used to construct a time-series-entity dual-dimensional dynamic constraint graph based on the dual time-series offsets, and to perform anomaly confidence calculation based on the time-series-entity dual-dimensional dynamic constraint graph to obtain anomaly detection results. The causality verification module is used to perform distributed causal consistency verification on the data corresponding to the anomaly detection results in order to determine the abnormal data to be repaired. The repair module is used to perform automatic repair on the abnormal data to be repaired at the lowest cost, and obtain the repaired data; The update module is used to write the repaired data back to the original data source and perform online incremental updates on the time-series-entity dual-dimensional dynamic constraint map based on the repaired data.

Images