Methods and systems for constructing and quality control of thematic databases based on multi-source disease control data fusion

Abstract

This invention discloses a method and system for constructing and quality controlling a thematic database of multi-source disease control data fusion. The method includes: establishing a unified data standard system to form an integrated disease control data resource through multi-granularity data fusion; calculating data dimension weights based on the information gain algorithm, using hierarchical clustering to divide the data into thematic sections, clarifying the core data dimensions and boundaries of each thematic database, and constructing a thematic association strength model to determine the thematic association relationship by combining data overlap and semantic similarity, thus forming a structured multi-dimensional disease control data thematic database; constructing a data quality assessment model based on Bayesian networks, using prior and posterior probabilities to calculate anomaly confidence to identify abnormal data, and optimizing quality control rule parameters through gradient descent algorithm to improve detection accuracy; and establishing a data quality traceability and closed-loop processing mechanism to dynamically optimize the rule system based on business feedback, expanding the data coverage and application dimensions of the thematic database.

Description

Technical Field

[0001] This invention belongs to the field of disease control data management technology, and more specifically, relates to a method and system for constructing and quality controlling a thematic database for multi-source disease control data fusion. Background Technology

[0002] With the continuous improvement of the public health prevention and control system, the sources of data in the field of disease control are becoming increasingly diversified, covering various types such as data reported by medical institutions, community monitoring data, laboratory testing data, and environmental monitoring data. These multi-source disease control data serve as the core support for public health decision-making, epidemic prevention and control early warning, and analysis of disease epidemiological patterns. Their effective integration and quality control are of great significance for improving the efficiency of disease control work and ensuring public health security.

[0003] Currently, multi-source disease control data faces numerous prominent problems in practical applications, severely restricting the realization of its value. First, disease control data from different sources lacks a unified standard system; data formats, coding rules, and granularity specifications differ significantly, making effective integration of multi-source data difficult, forming data silos, and failing to provide comprehensive and unified data support for disease control operations. Second, existing data organization methods are fragmented, lacking scientific thematic division based on disease control operational needs, resulting in unclear data relationships. This leads to low efficiency for staff when retrieving and accessing relevant data, making it difficult to quickly obtain targeted data for operational use.

[0004] Meanwhile, there are significant shortcomings in data quality control during the multi-source data fusion process. Due to the complexity of data sources and the different collection processes, problems such as data anomalies, missing data, and logical contradictions are prone to occur throughout the fusion process. Existing quality detection methods mostly use fixed rules for judgment, lacking adaptive adjustment capabilities and failing to adapt to the dynamic distribution characteristics of disease control data. This results in insufficient accuracy in detecting abnormal data, affecting the reliability and usability of the data.

[0005] Furthermore, existing technologies lack effective rule optimization and topic expansion mechanisms, making it impossible to adjust data processing rules in a timely manner according to the dynamic changes in disease control operations. The coverage and application dimensions of the topic database are insufficient to meet the growing needs of disease control operations, thus failing to fully realize the application value of multi-source disease control data. Summary of the Invention

[0006] This invention aims to address the practical problems of inconsistent standards, difficulties in data integration, chaotic thematic organization, and insufficient accuracy in data quality testing of multi-source disease control data. By constructing a standardized integration, thematic division, and intelligent quality control system, it improves the efficiency and reliability of disease control data integration, and provides accurate and efficient data support for public health prevention and control.

[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, as a first aspect of this invention, the present invention provides a method for constructing and quality-controlling a thematic database for multi-source disease control data fusion, comprising: S1. Establish a unified data standard system and adopt a hierarchical data fusion strategy to form an integrated disease control data resource; S2. Calculate the weight values ​​of each data dimension based on the information gain algorithm, and divide the integrated disease control data resources into thematic sections using hierarchical clustering to define the core data dimensions and boundaries of each thematic database. At the same time, construct a thematic association strength model, and determine the thematic association relationship by calculating the data overlap and semantic similarity between thematic sections, thus forming a structured multi-dimensional disease control data thematic database. S3. A data quality assessment model is constructed based on Bayesian networks. The confidence level of data anomalies is calculated by prior and posterior probabilities, and abnormal data in the entire process is automatically identified and integrated. The gradient descent algorithm is used to optimize the parameters of quality control rules and improve the accuracy of anomaly detection. S4. Establish a data quality issue tracing and closed-loop processing mechanism, dynamically optimize the core rule system based on business feedback, and expand the data coverage and application dimensions of the thematic database.

[0008] Furthermore, the unified data standard system in S1 is constructed based on the core attributes and application scenarios of disease control business data, and the specific process is as follows: First, we need to sort out the core data fields and business meanings of various data sources, including public health monitoring, laboratory testing, and internal management, and clarify the definition, data type, field length, and constraint rules of data elements; Then, a unified value domain dictionary is established to standardize and regulate the values ​​of key data items, including disease names, test items, and institution types. Finally, a cross-data source encoding mapping table is constructed, and a semantic matching algorithm is used to associate and bind synonymous codes in different systems to achieve consistency in field semantics and data format for multiple types of data.

[0009] Furthermore, the hierarchical data fusion strategy in S1 is specifically as follows: Based on data granularity, the data layer is divided into a basic data layer, a related data layer, and a fused data layer. The basic data layer performs field-level standardization on single-source data, eliminating invalid values ​​and duplicates. The associated data layer uses a unique identifier field as its core and employs a cosine similarity algorithm to match cross-source data entities and establish a mapping between data relationships. The fusion data layer employs a weighted fusion algorithm to integrate heterogeneous data from the same source in the related data layer. Weight coefficients are assigned based on the reliability of the data source, calculated using the following formula: ,in For the first Weight coefficients of each data source, For the first Credibility rating of each data source To determine the total number of data sources involved in the fusion, a unified fusion data result is obtained through weighted calculation, ultimately forming an integrated disease control data resource with consistent structure and unified semantics.

[0010] Furthermore, the process of thematically dividing the integrated disease control data resources using hierarchical clustering in S2 is as follows: First, using the integrated disease control data resources as clustering objects, a data correlation matrix is ​​constructed based on the discriminant values ​​of each data dimension calculated using the information gain algorithm. Then, the correlation between any two data objects is calculated using the improved Mahalanobis distance formula, as shown in the formula below. ,in For data objects and The degree of correlation, , These are the multidimensional feature vectors of two data objects. This is the inverse of the covariance matrix of the integrated disease control data resources, used to eliminate the influence of correlation between data dimensions. This is the average information gain across all data dimensions for the two data objects, used to enhance the contribution of high-discrimination dimensions to the correlation calculation; Then, a bottom-up aggregation strategy is adopted to merge two data clusters whose correlation meets a preset threshold into a new cluster. The feature vector of the new cluster is calculated by the mean of the feature vectors of all data objects in the cluster. This aggregation process is repeated until the correlation between all data clusters is lower than the preset threshold. Finally, based on the clustering results and combined with the core scenario requirements in disease control business, including public health monitoring, laboratory testing, and health management, the core data dimensions and boundaries of each thematic database are defined to ensure that the data in each thematic database has a high degree of business relevance and data independence.

[0011] Furthermore, the construction process of the topic association strength model in S2 is as follows: Using the disease control data themes divided by hierarchical clustering as the calculation objects, the data overlap and semantic similarity between themes are calculated separately. These two values ​​are then fused to determine the thematic relationships. Data overlap reflects the degree of overlap of actual data objects between themes, while semantic similarity characterizes the degree of correlation in the business content between themes. The calculation formula is as follows: in This represents the correlation strength value between topic database A and topic database B. This represents the number of data objects shared by topic database A and topic database B. This represents the total number of duplicate data objects between subject database A and subject database B. For topic database A in the first Feature values ​​on each data dimension; For topic database B in the first Feature values ​​on each data dimension; This refers to the total number of data dimensions after standardization.

[0012] Furthermore, the calculation process for the data anomaly confidence level in S3 is as follows: Using integrated disease control data as the analysis object, this paper integrates evidence theory and Bayesian networks to construct an improved data quality assessment model. First, the prior probabilities of normal and abnormal states of the data are obtained based on historical standardized data statistics. At the same time, basic probability assignment functions for corresponding states are constructed for multi-dimensional quality characteristics. Then, Dempster's combination rule is used to perform conflict resolution and fusion operations on multi-source quality characteristic evidence to obtain a global evidence confidence result. Subsequently, the global evidence confidence and Bayesian posterior probability are weighted and fused to calculate the final improved anomaly confidence.

[0013] Furthermore, the quality control rules in S3 take the anomaly confidence level of the fused data, data integrity indicators, and topic association matching degree as the core judgment elements, and the judgment formula is: in This is the comprehensive quality control judgment value. The data anomaly confidence score calculated for the Bayesian network. The number of valid fields in a single piece of merged data. This represents the total number of standard fields corresponding to this data entry. This is the numerical value that matches the data with the features of the subject database to which it belongs. The data was determined to be abnormal at that time. A fixed judgment threshold is obtained based on the historical distribution statistics of integrated disease control data.

[0014] Furthermore, the process of optimizing the quality control rule parameters using the gradient descent algorithm in S3 is as follows: Using standardized, integrated disease control data as training samples, and a comprehensive judgment threshold based on quality control rules... Construct a loss function for the parameters to be optimized. ,in The value of the loss function. The number of normal data points that were mistakenly identified as abnormal. The total number of data detected. This represents the number of undetected abnormal data. This represents the total number of actual abnormal data. The partial derivative of the loss function with respect to the threshold is calculated using the gradient descent algorithm. According to the iterative formula Update parameters, where For the first The threshold for the next iteration. For the first The threshold for the next iteration. Let be the iteration step size, and , The distribution variance of abnormal confidence in integrated disease control data is calculated; the iteration continues until the loss function converges to its minimum value, thus completing the adaptive optimization of the quality control rule parameters.

[0015] As a second aspect of the present invention, a thematic database construction and quality control system for multi-source disease control data fusion is also provided, comprising: The data standard system construction unit is used to establish a unified data standard system and adopt a hierarchical data fusion strategy to form an integrated disease control data resource. The thematic division and association modeling unit is used to calculate the weight values ​​of each data dimension based on the information gain algorithm, and to divide the integrated disease control data resources into thematic sections using hierarchical clustering, defining the core data dimensions and boundaries of each thematic database. At the same time, a thematic association strength model is constructed to determine the thematic association relationship by calculating the data overlap and semantic similarity between thematic sections, forming a structured multi-dimensional disease control data thematic database. The anomaly identification and quality control optimization unit is used to build a data quality assessment model based on Bayesian networks. It calculates the confidence level of data anomalies through prior and posterior probabilities, automatically identifies and integrates abnormal data in the entire process, and uses the gradient descent algorithm to optimize the parameters of quality control rules to improve the accuracy of anomaly detection. Thematic extension units are used to establish a data quality issue tracing and closed-loop processing mechanism, dynamically optimize the core rule system based on business feedback, and expand the data coverage and application dimensions of the thematic library.

[0016] As a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor as described in any one of the claims: a method for constructing and quality controlling a thematic library of multi-source disease control data fusion.

[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The method for constructing and quality controlling multi-source disease control data fusion thematic databases of this invention establishes a unified data standard system and adopts a hierarchical data fusion strategy to achieve standardized integration of multi-source heterogeneous disease control data, forming an integrated disease control data resource. This provides a unified data foundation for subsequent thematic division and quality assessment, solving the problems of inconsistent formats and granularities in multi-source disease control data, and ensuring the integrity and consistency of data fusion. Based on the information gain algorithm, the weight values ​​of each data dimension are calculated, accurately identifying core data dimensions that have a critical impact on disease control operations, providing a quantitative basis for thematic division. Then, hierarchical clustering is used to thematically divide the integrated disease control data resource, clearly defining the core data dimensions and boundaries of each thematic database. Simultaneously, a thematic association strength model is constructed, and thematic association relationships are determined by calculating the data overlap and semantic similarity between themes, forming a structured multi-dimensional disease control data thematic database. This improves the organization and retrieval of disease control data, facilitating accurate data retrieval in business scenarios.

[0018] 2. The multi-source disease control data fusion thematic library construction and quality control method of this invention dynamically iterates and optimizes the core rule system based on feedback from actual disease control operations. This ensures that data fusion rules, thematic division rules, and quality control rules continuously match the actual usage requirements of business scenarios. The rule optimization process closely relies on the already constructed integrated data and thematic library structure, ensuring the consistency of the overall technical process. Based on rule optimization, the data inclusion scope and business application dimensions of the thematic library are gradually expanded, enabling the thematic library to carry more types of disease control data and support more segmented business scenarios. This strengthens the practical application capabilities of the multi-source disease control data thematic library and allows the overall technical solution to continuously adapt to business changes, achieving long-term stable operation of the thematic library construction and quality control system.

[0019] 3. The method for constructing and controlling thematic databases for multi-source disease control data fusion in this invention constructs a data quality assessment model based on Bayesian networks and calculates the confidence level of data anomalies using prior and posterior probabilities. This enables quantitative quality assessment of data throughout the fusion process and automatic identification of abnormal data. Compared to traditional rule-based detection methods, this method better aligns with the probability distribution characteristics of disease control data, improving the objectivity and accuracy of anomaly identification. Furthermore, a gradient descent algorithm is used to optimize the parameters of the quality control rules, aiming to minimize the sum of the false positive and false negative rates for anomaly detection. The core judgment thresholds of the quality control rules are iteratively updated, establishing a dynamic adjustment mechanism adapted to the data distribution characteristics. This further improves the accuracy of anomaly detection, avoids the poor adaptability of fixed thresholds, and ensures the reliability of the fused disease control data. Attached Figure Description

[0020] Figure 1This is a flowchart illustrating the method for constructing and quality controlling a thematic database for multi-source disease control data fusion according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the system units in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1 Please refer to Figure 1 This embodiment 1 provides a method for constructing and quality controlling a thematic database based on multi-source disease control data fusion, including: S1. Establish a unified data standard system and adopt a hierarchical data fusion strategy to form an integrated disease control data resource; S2. Calculate the weight values ​​of each data dimension based on the information gain algorithm, and divide the integrated disease control data resources into thematic sections using hierarchical clustering to define the core data dimensions and boundaries of each thematic database. At the same time, construct a thematic association strength model, and determine the thematic association relationship by calculating the data overlap and semantic similarity between thematic sections, thus forming a structured multi-dimensional disease control data thematic database. S3. A data quality assessment model is constructed based on Bayesian networks. The confidence level of data anomalies is calculated by prior and posterior probabilities, and abnormal data in the entire process is automatically identified and integrated. The gradient descent algorithm is used to optimize the parameters of quality control rules and improve the accuracy of anomaly detection. S4. Establish a data quality issue tracing and closed-loop processing mechanism, dynamically optimize the core rule system based on business feedback, and expand the data coverage and application dimensions of the thematic database.

[0023] This embodiment 1 further elaborates on the above steps.

[0024] (1) Construction of data standard system In the course of disease control and prevention work, multi-source data often comes from different information systems. These systems have different data formats, coding rules, and field definitions, easily creating data silos. This prevents effective data integration and utilization, hindering the full realization of data's supporting role. Therefore, it is necessary to first establish a unified data standard system, then adopt a layered data fusion strategy, and ultimately form an integrated disease control and prevention data resource, providing a foundation for subsequent thematic construction and quality control.

[0025] The unified data standard system is built upon the core attributes and practical application scenarios of disease control business data, ensuring that all types of data have unified specifications and laying the foundation for data integration. Specifically, it first comprehensively reviews various data sources, covering major business areas such as public health monitoring, laboratory testing, and internal management, clarifying the core data fields of each data source and their corresponding business meanings. Simultaneously, it clearly defines the specific definition, data type, field length, and related constraints of each data element, avoiding data chaos caused by ambiguous field definitions. Based on this, a unified value domain dictionary is established, clarifying the standardized value range and expression specifications for key data items such as disease names, testing items, and institution types, ensuring consistent values ​​and semantics for the same type of data. Finally, a cross-data source coding mapping table is constructed. Through semantic matching algorithms, synonymous codes with the same meaning but different codes in different information systems are associated and bound, completely resolving inconsistencies in field semantics and data format among various types of data, achieving data standardization and unification.

[0026] The hierarchical data fusion strategy achieves efficient integration of multi-source data through hierarchical processing, ultimately forming a unified disease control data resource with consistent structure and semantics. This strategy is divided into three levels based on data granularity: a basic data layer, a related data layer, and a fused data layer, advancing data integration work in a layered manner.

[0027] Based on data granularity, the data is divided into a basic data layer, a related data layer, and a fused data layer. The basic data layer mainly performs field-level standardization processing on data from a single source, checking and eliminating invalid values ​​and duplicates in the data one by one to ensure the accuracy and standardization of the single-source data itself. The associated data layer takes the unique identifier field in the data as the core and uses the cosine similarity algorithm to accurately match data entities from different sources, establish an association mapping relationship between cross-source data, break down the barriers between different data sources, and realize the interconnection of data. The fusion data layer employs a weighted fusion algorithm to integrate heterogeneous data from the same source in the related data layer. Weight coefficients are assigned based on the reliability of the data source, calculated using the following formula: ,in For the first Weight coefficients of each data source, For the first Credibility rating of each data source To determine the total number of data sources involved in the fusion, a unified fusion data result is obtained through weighted calculation, ultimately forming an integrated disease control data resource with consistent structure and unified semantics.

[0028] For example: Suppose a disease control center needs to integrate three types of core data (outpatient case reporting data, laboratory nucleic acid test data, and community follow-up data). Using a layered integration strategy, the specific implementation process is as follows: Basic Data Layer (Single-Source Standardization Processing): Standardization and cleaning are performed on the three types of data to ensure uniform format and the absence of invalid data. Outpatient case reporting data is standardized with uniform formatting for fields such as "Case ID, Onset Date, Symptom Type," and invalid records with errors or abnormal dates are removed. Laboratory nucleic acid testing data has standardized fields for "Test ID, Test Result, Test Time," and standardized result descriptions (retaining only "Positive," "Negative," and "Suspected"). Community follow-up data has standardized fields for "Follow-up ID, Follow-up Time, Follow-up Result," and duplicate follow-up records are deleted. Upon completion, each type of data conforms to a unified standard, laying the foundation for cross-source fusion.

[0029] Association and Fusion Layer (Cross-Source Data Matching): Using "Case ID" as the core association identifier, the cosine similarity algorithm is used to accurately match the "Case ID" in outpatient case data with the "corresponding Case ID" in laboratory test data, establishing an association between the two. At the same time, the "Follow-up ID" in community follow-up data is associated with the "Case ID", ensuring that the outpatient reports, nucleic acid tests, and community follow-up data of the same case correspond one-to-one, breaking the isolation of the three types of data, realizing data interoperability, and avoiding data disconnection.

[0030] Unified Integration Layer (Forming Usable Data): This layer integrates the three types of data after association, eliminating duplicate indicators (such as duplicate basic case information) and retaining core effective data (basic case information, test results, and follow-up status). The data granularity is unified at the "daily level," aligning with the daily monitoring needs of disease control. The integrated data structure is unified and logically coherent, and can be directly used for subsequent epidemic analysis and trend assessment. Simultaneously, it provides clear and reliable basic data for quality control, ensuring that the data can directly support disease control operations.

[0031] (2) Thematic division and correlation modeling In the application of multi-source disease control data, the inconsistent data formats and unclear business objectives from different sources make it difficult to effectively integrate and utilize the data, thus failing to fully leverage its supporting role in disease control work. Therefore, it is necessary to achieve standardized data utilization through reasonable classification and management. Based on this, this paper first uses the obtained integrated disease control data resources as clustering objects. Based on the discriminancy values ​​of each data dimension calculated using the information gain algorithm, a data correlation matrix is ​​constructed. Then, an improved Mahalanobis distance formula is used to calculate the correlation between any two data objects. The formula is as follows: ,in For data objects and The degree of correlation, , These are the multidimensional feature vectors of two data objects. This is the inverse of the covariance matrix of the integrated disease control data resources, used to eliminate the influence of correlation between data dimensions. This is the average information gain across all data dimensions for the two data objects, used to enhance the contribution of high-discrimination dimensions to the correlation calculation; Then, a bottom-up aggregation strategy is adopted to merge two data clusters whose correlation meets a preset threshold into a new cluster. The feature vector of the new cluster is calculated by the mean of the feature vectors of all data objects in the cluster. This aggregation process is repeated until the correlation between all data clusters is lower than the preset threshold. Finally, based on the clustering results and combined with the core scenario requirements in disease control business, including public health monitoring, laboratory testing, and health management, the core data dimensions and boundaries of each thematic database are defined to ensure that the data in each thematic database has a high degree of business relevance and data independence.

[0032] After completing the thematic division, a thematic association strength model was constructed to clarify the relationships between the themes. This model uses each disease control data theme after hierarchical clustering as the calculation object, calculating the data overlap and semantic similarity between the themes separately, and then fusing these two metrics to determine the thematic associations. Data overlap reflects the degree of overlap of actual data objects between the themes, while semantic similarity characterizes the degree of association of business content between the themes. The calculation formula is as follows: in This represents the correlation strength value between topic database A and topic database B. This represents the number of data objects shared by topic database A and topic database B. This represents the total number of duplicate data objects between subject database A and subject database B. For topic database A in the first Feature values ​​on each data dimension; For topic database B in the first Feature values ​​on each data dimension; This refers to the total number of data dimensions after standardization.

[0033] Through the above series of operations, a multi-dimensional disease control data thematic library with a clear structure and logical coherence is finally formed, realizing the orderly organization of disease control data, solving the problem of messy and inefficient use of multi-source data, and providing standardized and reliable data support for the development of disease control business.

[0034] For example, taking a municipal-level CDC as an example, the center needs to process three types of core data on a daily basis: infectious disease monitoring data, laboratory testing data, and health record data of key populations. These three types of data come from different sources and have different formats, making them impossible to integrate and utilize directly. Moreover, the data is scattered and disorganized, which greatly inconveniences epidemic assessment and epidemiological investigation. Therefore, it is necessary to fully utilize the value of the data through standardized processing and classification.

[0035] In the specific implementation process, the core fields of the three types of data were first identified to clarify the business meaning and usage scenarios of each type of data. The importance of each data dimension was determined using an information gain algorithm. Then, hierarchical clustering was used to group similar business data into one category, creating three thematic databases: infectious disease surveillance, laboratory testing, and key population management. Each thematic database corresponds to a specific business scenario, ensuring accurate matching between data and business needs. Simultaneously, semantic matching was used to establish relationships between the themes, enabling collaborative access to data from different themes. For example, data from the infectious disease surveillance and laboratory testing themes can be shared, avoiding data silos. Through this process, staff can quickly retrieve comprehensive data for the same case during epidemiological investigations without having to sift through each case individually, significantly shortening data retrieval time, reducing manual processing workload, and preventing delays in judgment due to data chaos, thereby effectively improving the efficiency and accuracy of epidemic prevention and control.

[0036] (3) Anomaly identification and quality control optimization Even after multi-source disease control data is integrated and thematically constructed, issues such as missing data, contradictions, and distortions may still occur during data collection, transmission, and integration. Traditional quality control methods with fixed thresholds are difficult to adapt to the dynamic changes in data distribution, easily leading to missed detections or misjudgments, affecting the reliability of subsequent business analysis. To achieve end-to-end data quality control, it is necessary to build an assessment and optimization mechanism that combines stability and adaptability, automatically identifying abnormal data and continuously optimizing judgment rules.

[0037] First, the Bayesian network nodes are linked to the disease control data quality dimensions (completeness, accuracy, consistency), and the basic prior probabilities of each node are obtained through historical normal sample statistics. At the same time, a basic probability assignment function is constructed based on evidence theory. The credibility of different data sources is quantified and allocated to form node-level evidence trust levels. and ,in The level of confidence in evidence that a node is in a normal state. The confidence level of evidence indicating that a node is in an abnormal state, and satisfies the following conditions: The remaining trust is allocated to unknown states to address the problem of data uncertainty. Secondly, regarding the data to be tested Based on the feature extraction results from the thematic database, the observed data under normal conditions are calculated for each quality dimension. and abnormal state conditional probability and Furthermore, the Dempster combination rule is used to fuse the multi-dimensional evidence trust scores to obtain the fused global evidence trust score. and The fusion formula is: ,in Is it a normal or abnormal state? , The basic probability assignment function for two independent pieces of evidence; Finally, the weighted global evidence confidence score and the Bayesian posterior probability are weighted and fused to construct an improved formula for calculating anomaly confidence. ,in The improved data anomaly confidence level has a value range of [0, 1]. and To integrate the weighting coefficients, and satisfy the following conditions: This coefficient is obtained through cross-validation optimization of historical samples and is used to balance the objectivity of Bayesian probability with the uncertainty handling capability of evidence theory.

[0038] Based on anomaly identification, a comprehensive quality control rule is established. This quality control rule uses the anomaly confidence level of the fused data, data integrity indicators, and topic relevance matching degree as core judgment elements. The judgment formula is as follows: in This is the comprehensive quality control judgment value. The data anomaly confidence score calculated for the Bayesian network. The number of valid fields in a single piece of merged data. This represents the total number of standard fields corresponding to this data entry. This is the numerical value that matches the data with the features of the subject database to which it belongs. The data was determined to be abnormal at that time. A fixed judgment threshold is obtained based on the historical distribution statistics of integrated disease control data.

[0039] Since fixed thresholds are difficult to adapt to the data distribution characteristics of different time periods, a gradient descent algorithm is further used to adaptively optimize the judgment threshold. For example, using standardized integrated disease control data as training samples, and a comprehensive judgment threshold based on quality control rules is used. Construct a loss function for the parameters to be optimized. ,in The value of the loss function. The number of normal data points that were mistakenly identified as abnormal. The total number of data detected. This represents the number of undetected abnormal data. This represents the total number of actual abnormal data. The partial derivative of the loss function with respect to the threshold is calculated using the gradient descent algorithm. According to the iterative formula Update parameters, where For the first The threshold for the next iteration. For the first The threshold for the next iteration. Let be the iteration step size, and , The distribution variance of abnormal confidence in integrated disease control data is calculated; the iteration continues until the loss function converges to the minimum value, and the optimal threshold is input into the dynamic threshold adjustment mechanism to complete the adaptive optimization of quality control rule parameters.

[0040] (4) Thematic extension As disease control data is used and expanded across various business scenarios over the long term, new quality issues and rule adaptation requirements will continuously emerge. Relying solely on one-time quality control is insufficient to guarantee the continuous availability of data and cannot meet the dynamic development requirements of public health operations. To ensure the long-term stable operation of the data system and its continuous adaptation to actual work, it is necessary to establish a complete data quality issue tracing and closed-loop processing mechanism. Simultaneously, core rules should be continuously optimized based on feedback from business operations, and the data coverage and application dimensions of the thematic databases should be expanded in parallel.

[0041] In terms of data quality management, a full-process traceability mechanism is established to record and locate the links, types, and causes of abnormal data occurrences. This clarifies whether the problem originates from the data collection, cross-source fusion, or topic matching stage, forming a standardized process for problem registration, verification, rectification, and review. This achieves closed-loop processing from problem discovery to problem resolution, preventing the recurrence of similar quality issues.

[0042] Based on feedback from actual operational use, core elements such as data fusion rules, the basis for topic division, and quality assessment parameters are dynamically adjusted to make the entire rule system more aligned with real-world business scenarios such as grassroots disease control, epidemic monitoring, and health management. While optimizing the rules, the coverage of the topic database is gradually expanded to include new business needs and data sources, supplementing data types not covered in existing topics and extending the database's application dimensions in areas such as risk warning, comprehensive analysis, and collaborative response. This ensures the entire data system has continuous iteration capabilities and can stably support the efficient operation of disease control services in the long term.

[0043] Example 2 Please refer to Figure 2 This embodiment 2 provides a thematic database construction and quality control system for multi-source disease control data fusion, including: The data standard system construction unit is used to establish a unified data standard system and adopt a hierarchical data fusion strategy to form an integrated disease control data resource. The thematic division and association modeling unit is used to calculate the weight values ​​of each data dimension based on the information gain algorithm, and to divide the integrated disease control data resources into thematic sections using hierarchical clustering, defining the core data dimensions and boundaries of each thematic database. At the same time, a thematic association strength model is constructed to determine the thematic association relationship by calculating the data overlap and semantic similarity between thematic sections, forming a structured multi-dimensional disease control data thematic database. The anomaly identification and quality control optimization unit is used to build a data quality assessment model based on Bayesian networks. It calculates the confidence level of data anomalies through prior and posterior probabilities, automatically identifies and integrates abnormal data in the entire process, and uses the gradient descent algorithm to optimize the parameters of quality control rules to improve the accuracy of anomaly detection. Thematic extension units are used to establish a data quality issue tracing and closed-loop processing mechanism, dynamically optimize the core rule system based on business feedback, and expand the data coverage and application dimensions of the thematic library.

[0044] Example 3 This embodiment 3 also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement any step of a method for constructing and quality controlling a thematic database of multi-source disease control data fusion.

[0045] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0046] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0047] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing and quality-controlling a thematic database based on multi-source disease control data fusion, characterized in that, include: S1. Establish a unified data standard system and adopt a hierarchical data fusion strategy to form an integrated disease control data resource; S2. Calculate the weight values ​​of each data dimension based on the information gain algorithm, and divide the integrated disease control data resources into thematic sections using hierarchical clustering to define the core data dimensions and boundaries of each thematic database. At the same time, construct a thematic association strength model, and determine the thematic association relationship by calculating the data overlap and semantic similarity between thematic sections, thus forming a structured multi-dimensional disease control data thematic database. S3. A data quality assessment model is constructed based on Bayesian networks. The confidence level of data anomalies is calculated by prior and posterior probabilities, and abnormal data in the entire process is automatically identified and integrated. The gradient descent algorithm is used to optimize the parameters of quality control rules and improve the accuracy of anomaly detection. S4. Establish a data quality issue tracing and closed-loop processing mechanism, dynamically optimize the core rule system based on business feedback, and expand the data coverage and application dimensions of the thematic database.

2. The method for constructing and quality controlling a thematic database of multi-source disease control data fusion according to claim 1, characterized in that, The unified data standard system in S1 is built based on the core attributes and application scenarios of disease control business data. The specific process is as follows: First, we need to sort out the core data fields and business meanings of various data sources, including public health monitoring, laboratory testing, and internal management, and clarify the definition, data type, field length, and constraint rules of data elements; Then, a unified value domain dictionary is established to standardize and regulate the values ​​of key data items, including disease names, test items, and institution types. Finally, a cross-data source encoding mapping table is constructed, and a semantic matching algorithm is used to associate and bind synonymous codes in different systems to achieve consistency in field semantics and data format for multiple types of data.

3. The method for constructing and quality controlling a thematic database of multi-source disease control data fusion according to claim 1, characterized in that, The hierarchical data fusion strategy in S1 is specifically as follows: Based on data granularity, the data layer is divided into a basic data layer, a related data layer, and a fused data layer. The basic data layer performs field-level standardization on single-source data, eliminating invalid values ​​and duplicates. The associated data layer uses a unique identifier field as its core and employs a cosine similarity algorithm to match cross-source data entities and establish a mapping between data relationships. The fusion data layer employs a weighted fusion algorithm to integrate heterogeneous data from the same source in the related data layer. Weight coefficients are assigned based on the reliability of the data source, calculated using the following formula: ,in For the first Weight coefficients of each data source, For the first Credibility rating of each data source To determine the total number of data sources involved in the fusion, a unified fusion data result is obtained through weighted calculation, ultimately forming an integrated disease control data resource with consistent structure and unified semantics.

4. The method for constructing and quality controlling a thematic database of multi-source disease control data fusion according to claim 1, characterized in that, The process of thematic division of integrated disease control data resources using hierarchical clustering in S2 is as follows: First, using the integrated disease control data resources as clustering objects, a data correlation matrix is ​​constructed based on the discriminant values ​​of each data dimension calculated using the information gain algorithm. Then, the correlation between any two data objects is calculated using the improved Mahalanobis distance formula, as shown in the formula below. ,in For data objects and The degree of correlation, , These are the multidimensional feature vectors of two data objects. This is the inverse of the covariance matrix of the integrated disease control data resources, used to eliminate the influence of correlation between data dimensions. This is the average information gain across all data dimensions for the two data objects, used to enhance the contribution of high-discrimination dimensions to the correlation calculation; Then, a bottom-up aggregation strategy is adopted to merge two data clusters whose correlation meets a preset threshold into a new cluster. The feature vector of the new cluster is calculated by the mean of the feature vectors of all data objects in the cluster. This aggregation process is repeated until the correlation between all data clusters is lower than the preset threshold. Finally, based on the clustering results and combined with the core scenario requirements in disease control business, including public health monitoring, laboratory testing, and health management, the core data dimensions and boundaries of each thematic database are defined to ensure that the data in each thematic database has a high degree of business relevance and data independence.

5. The method for constructing and quality controlling a thematic database for multi-source disease control data fusion according to claim 1, characterized in that, The construction process of the topic association strength model in S2 is as follows: Using the disease control data themes divided by hierarchical clustering as the calculation objects, the data overlap and semantic similarity between themes are calculated separately. These two values ​​are then fused to determine the thematic relationships. Data overlap reflects the degree of overlap of actual data objects between themes, while semantic similarity characterizes the degree of correlation in the business content between themes. The calculation formula is as follows: in This represents the correlation strength value between topic database A and topic database B. This represents the number of data objects shared by topic database A and topic database B. This represents the total number of duplicate data objects between subject database A and subject database B. For topic database A in the first Feature values ​​on each data dimension; For topic database B in the first Feature values ​​on each data dimension; This refers to the total number of data dimensions after standardization.

6. The method for constructing and quality controlling a thematic database of multi-source disease control data fusion according to claim 1, characterized in that, The calculation process for the data anomaly confidence level in S3 is as follows: Using integrated disease control data as the analysis object, this paper integrates evidence theory and Bayesian networks to construct an improved data quality assessment model. First, the prior probabilities of normal and abnormal states of the data are obtained based on historical standardized data statistics. At the same time, basic probability assignment functions for corresponding states are constructed for multi-dimensional quality characteristics. Then, Dempster's combination rule is used to perform conflict resolution and fusion operations on multi-source quality characteristic evidence to obtain a global evidence confidence result. Subsequently, the global evidence confidence and Bayesian posterior probability are weighted and fused to calculate the final improved anomaly confidence.

7. The method for constructing and quality control of a thematic database for multi-source disease control data fusion according to claim 1, characterized in that, The quality control rules in S3 use the anomaly confidence level of fused data, data integrity indicators, and topic correlation matching degree as core judgment elements, and the judgment formula is: in This is the comprehensive quality control judgment value. The data anomaly confidence score calculated for the Bayesian network. The number of valid fields in a single piece of merged data. This represents the total number of standard fields corresponding to this data entry. This is the numerical value that matches the data with the features of the subject database to which it belongs. The data was determined to be abnormal at that time. A fixed judgment threshold is obtained based on the historical distribution statistics of integrated disease control data.

8. The method for constructing and quality control of a thematic database for multi-source disease control data fusion according to claim 1, characterized in that, The process of optimizing the quality control rule parameters using the gradient descent algorithm in S3 is as follows: Using standardized, integrated disease control data as training samples, and a comprehensive judgment threshold based on quality control rules... Construct a loss function for the parameters to be optimized. ,in The value of the loss function. The number of normal data points that were mistakenly identified as abnormal. The total number of data detected. This represents the number of undetected abnormal data. This represents the total number of actual abnormal data. The partial derivative of the loss function with respect to the threshold is calculated using the gradient descent algorithm. According to the iterative formula Update parameters, where For the first The threshold for the next iteration. For the first The threshold for the next iteration. Let be the iteration step size, and , The distribution variance of abnormal confidence in integrated disease control data is calculated; the iteration continues until the loss function converges to its minimum value, thus completing the adaptive optimization of the quality control rule parameters.

9. A thematic database construction and quality control system for multi-source disease control data fusion, characterized in that, include: The data standard system construction unit is used to establish a unified data standard system and adopt a hierarchical data fusion strategy to form an integrated disease control data resource. The thematic division and association modeling unit is used to calculate the weight values ​​of each data dimension based on the information gain algorithm, and to divide the integrated disease control data resources into thematic sections using hierarchical clustering, defining the core data dimensions and boundaries of each thematic database. At the same time, a thematic association strength model is constructed to determine the thematic association relationship by calculating the data overlap and semantic similarity between thematic sections, forming a structured multi-dimensional disease control data thematic database. The anomaly identification and quality control optimization unit is used to build a data quality assessment model based on Bayesian networks. It calculates the confidence level of data anomalies through prior and posterior probabilities, automatically identifies and integrates abnormal data in the entire process, and uses the gradient descent algorithm to optimize the parameters of quality control rules to improve the accuracy of anomaly detection. Thematic extension units are used to establish a data quality issue tracing and closed-loop processing mechanism, dynamically optimize the core rule system based on business feedback, and expand the data coverage and application dimensions of the thematic library.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-8: a method for constructing and quality controlling a thematic library of multi-source disease control data fusion.

Images