A public health data acquisition and processing system

Abstract

The present invention discloses a public health data acquisition and processing system, which includes 1. the following stages: a data preparation stage, a feature engineering stage, and a model assessment and evaluation stage. The data preparation stage includes the following steps: S1, selecting diabetes as a project example and starting the project; S2, selecting a business information system and a data resource center to obtain business data for a diabetes follow-up scenario; S3, assigning reliability labels to the obtained data with clear data reliability; S4, collecting and classifying reliability labels, classifying the reliability data separately, and separating it from unprocessed data; S5, extracting some reliability labels from the screened data; S6, extracting the data type, data structure, and data form in the target data, and checking for missing values, abnormal values, and duplicate values ​​in the data; the system has the advantage of being able to classify and describe the data and reduce the subsequent analysis time.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and processing, and in particular to a public health data acquisition and processing system. Background Art

[0002] With the development of information technology and the advent of the big data era, more and more scientific research has begun to favor the use of existing data. At the same time, the number of integrated studies across multiple disciplines and fields has gradually increased, requiring the use of data from multiple disciplines and sources in the research. Before the research is carried out, the reliability of the data to be included in the study should be assessed, and corresponding measures should be taken based on the assessment results to improve the authenticity and accuracy of the research results. Data reliability refers to the degree to which the data is complete, consistent, accurate, trustworthy and reliable, and the extent to which these characteristics are maintained throughout the data life cycle. The reliability of data will be reduced by the existence of bias in the data. Common data biases mainly include selection bias, information bias, and confounding bias, and even include the fabrication and tampering of data. If low-reliability data is used in scientific research, the research results will deviate from the actual situation, greatly reducing the value of the research results. In the big data era, data reliability assessment is very important. It is related to the success or failure of the research and the value of the results. Before conducting scientific research, using scientific methods to accurately assess the reliability of the data to be included in the research is a necessary step in scientific research in the big data era.

[0003] In the field of public health, existing data reliability assessment methods are mainly divided into three categories: rule-based assessment methods, content-based assessment methods, and statistics-based assessment methods:

[0004] The rule-based evaluation method mainly sets up a rule base and uses the rules in the rule base to verify the data, and evaluates the reliability of the data through the verification results. Although the rule-based evaluation method can evaluate the entire data and discover data reliability issues caused by some information biases, the evaluation depth is relatively shallow and it is powerless to fabricate or tamper with data that meets the verification rules. In addition, the formulation of the rule base requires considerable familiarity with the business field where the data is located, which makes this method have a high implementation threshold when evaluating the reliability of cross-industry data.

[0005] Content-based assessment methods primarily use data from other sources to cross-validate the content of the data being assessed, thereby assessing the reliability of the data. Commonly used data sources include data collected through telephone / home visits, data obtained from reviewing original medical histories, etc. Content-based assessment methods can provide in-depth assessments, have a better assessment effect on information bias, and can also provide clues to selection bias and confounding bias to a certain extent. However, obtaining data from other sources usually requires a lot of time, energy, and financial costs. Sometimes, data from other sources may be unavailable, making it difficult to conduct reliability assessments on the entire data set with this method. It generally needs to be used in combination with sampling methods, and the reliability of the data obtained from the assessment may be biased.

[0006] Statistical evaluation methods primarily assess the reliability of data by calculating and analyzing the overall statistical results and distribution of the data being evaluated. For example, whether the last digit of the blood pressure value conforms to a random distribution, whether the male-female ratio in the data deviates significantly from the male-female ratio of the entire population, etc. Statistical evaluation methods are effective in assessing selection bias and information bias, and can also evaluate the entire data set. However, this method has certain requirements for the amount of data to be evaluated, and can only produce reliability assessment results for the overall data, and cannot form independent evaluation results for each data point.

[0007] Therefore, the existing public health data reliability assessment methods have some shortcomings and cannot fully meet the reliability assessment needs of massive data in the big data era. It is necessary to explore a new method to make up for the shortcomings of the existing methods, more accurately assess the reliability of public health data, and find more clues to data reliability issues. Among them, the collection and processing of public health data lack accuracy and contain errors. Therefore, we use diabetes follow-up data as a research example and propose a public health data acquisition and processing system. Summary of the Invention

[0008] The purpose of the present invention is to solve the shortcomings of the existing technology and to propose a public health data acquisition and processing system.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A public health data acquisition and processing system includes the following stages: a data preparation stage, a feature engineering stage, and a model assessment and evaluation stage. The data preparation stage includes the following steps:

[0011] S1, select diabetes as a project example and start the project;

[0012] S2, select the business information system and data resource center to obtain business data for the diabetes follow-up scenario;

[0013] S3, assigns reliability labels to the obtained data whose reliability has been confirmed;

[0014] S4, collects and classifies reliability labels, classifies reliability data separately, and separates them from unprocessed data;

[0015] S5, partial reliability labels for data extraction after screening;

[0016] S6, extracting the data type, data structure and data form of the target data, and checking the data for missing values, abnormal values ​​and duplicate values;

[0017] S7 extracts some key features and calculates statistical values, including maximum, minimum, average, and standard deviation, and displays them through visual graphs such as scatter plots, bar charts, and line charts;

[0018] S8, observe and understand the distribution of the visualization graph, determine whether the value conforms to a certain distribution pattern and whether there is a trend of concentration or dispersion;

[0019] S9, counting the data volume of each label category, and checking whether the data volume in each label category is unbalanced.

[0020] The present invention has the following beneficial effects:

[0021] In the present invention, business data of diabetes follow-up scenarios obtained from the business information system and the data resource center are labeled and classified, and unclassified business data are analyzed and analyzed in interval distribution. The corresponding classification labels can be extracted so that a more detailed analysis can be carried out in the subsequent processing process, which greatly reduces the difficulty of data analysis, eliminates some reasonable data, and only analyzes unreasonable data, reducing the time and workload of data analysis. DETAILED DESCRIPTION

[0022] The technical solutions of the present invention will be described clearly and completely below with reference to the embodiments.

[0023] Example 1

[0024] A public health data acquisition and processing system includes the following stages: a data preparation stage, a feature engineering stage, and a model assessment and evaluation stage. The data preparation stage includes the following steps:

[0025] S1, select diabetes as a project example and start the project;

[0026] S2, select the business information system and data resource center to obtain business data for the diabetes follow-up scenario;

[0027] S3, assigns reliability labels to the obtained data whose reliability has been confirmed;

[0028] S4, collects and classifies reliability labels, classifies reliability data separately, and separates them from unprocessed data;

[0029] S5, partial reliability labels for data extraction after screening;

[0030] S6, extracting the data type, data structure and data form of the target data, and checking the data for missing values, abnormal values ​​and duplicate values;

[0031] S7 extracts some key features and calculates statistical values, including maximum, minimum, average, and standard deviation, and displays them through visual graphs such as scatter plots, bar charts, and line charts;

[0032] S8, observe and understand the distribution of the visualization graph, determine whether the value conforms to a certain distribution pattern and whether there is a trend of concentration or dispersion;

[0033] S9, counting the data volume of each label category, and checking whether the data volume in each label category is unbalanced.

[0034] The raw data obtained at this time is divided into several categories, namely reliability data and some key feature labels divided according to data type, structure and form.

[0035] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A public health data acquisition and processing system, characterized in that: The process includes the following stages: data preparation, feature engineering, and model evaluation and assessment. The data preparation stage includes the following steps: S1, select diabetes as a project example and start the project; S2, select the business information system and data resource center to obtain business data for the diabetes follow-up scenario; S3, assigns reliability labels to the obtained data whose reliability has been confirmed; S4, collects and classifies reliability labels, classifies reliability data separately, and separates them from unprocessed data; S5, partial reliability labels for data extraction after screening; S6, extracting the data type, data structure and data form of the target data, and checking the data for missing values, abnormal values ​​and duplicate values; S7 extracts some key features and calculates statistical values, including maximum, minimum, average, and standard deviation, and displays them through visual graphs such as scatter plots, bar charts, and line charts; S8, observe and understand the distribution of the visualization graph, determine whether the value conforms to a certain distribution pattern and whether there is a trend of concentration or dispersion; S9, counting the data volume of each label category, and checking whether the data volume in each label category is unbalanced.