Packaging method for realizing medical data governance based on Kettle
The encapsulation of Kettle modules for medical data governance simplifies the assembly of application modules, addressing high development costs and skill dependencies, enhancing efficiency and system availability.
Patent Information
- Application Number
- CN202411378545.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-12-27
AI Technical Summary
Existing medical data governance solutions based on Kettle face high development costs and require specialized personnel, and are inefficient when business needs change.
A packaging method that encapsulates Kettle modules according to medical data governance scenarios, combining business and process modules to reduce dependency on specialized skills, allowing developers to focus on their respective domains.
Enables efficient data governance by simplifying the assembly of application modules, reducing dependency on specialized skills, and enhancing system availability and implementation efficiency.
Smart Images
Figure CN119201055A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer software technology, and relates to medical data processing technology, and in particular to a packaging method for medical data governance based on Kettle. Background Art
[0002] There are two existing medical data governance solutions: one is to develop data governance processes directly based on Kettle. The advantages of this method are that it is highly flexible and can be customized close to the underlying layer. However, the development cost of native Kettle is high, and it requires personnel with certain data development experience to operate. The other is to customize the packaging based on Kettle and develop system applications on Kettle. However, using a more customized approach, a large amount of system transformation costs are required when business needs and application scenarios change slightly. Therefore, it is necessary to improve it. Summary of the invention
[0003] In order to address the deficiencies of the above-mentioned prior art, the present application provides a packaging method for implementing medical data governance based on Kettle, which encapsulates and combines Kettle modules according to medical data governance scenarios. When applied, data governance can be achieved by simply assembling the corresponding application modules. Kettle processes are deeply bound to medical standard data elements and business models, thereby reducing the dependence of data developers on medical business development capabilities, and at the same time reducing the dependence of medical business developers on data development capabilities.
[0004] In order to achieve the above object, the present invention adopts the following technologies: The encapsulation method for medical data governance based on Kettle includes: Build data models based on health standard data elements to form business modules; Build process modules through Kettle; Combine business modules and process modules to build business governance processes; the process module is used to clarify the governance process, and the business module is used to fill in content to clarify specific governance content.
[0005] Furthermore, building a data model based on health standard data elements to form a business module includes: Constructing a medical business model: inheriting data elements according to reference standards, and combining multiple data elements into a medical business model; wherein the data type, representation format, and allowed value of the data element are used to describe the basic metadata of a medical information, and the reference standard is one or more of the national standard health information data element, local medical business standards, and hospital business standards; the medical business model is used to generate the table structure of the corresponding database for data storage; Data verification configuration processing: By parsing the data elements to extract the corresponding verification rules to form the data verification configuration, the data verification configuration can be referenced by the model fields; Model relationship configuration processing: Based on the constructed medical business model, supplement the correspondence between models to clarify the business relevance between businesses; Model mapping configuration processing: Map models with the same business meaning between models according to business requirements and generate model mapping configurations; and supplement the generated model mapping configurations with the mapping relationships required for mapping conversion for subsequent data cleaning.
[0006] Furthermore, building a process module through Kettle includes building the following process units separately: Data collection unit: used to read the medical business model and model mapping configuration from the business module to complete data collection from the source table to the target table. The source table and the target table are pre-bound to the medical business model and are consistent with the structure of the bound medical business model respectively; Data verification unit: used to read data verification configuration from the business module to complete data verification; Business verification unit: used to read the model relationship configuration from the business module to complete the business relationship verification; Data cleaning unit: used to read the model relationship configuration from the business module according to the fields and configurations that need to be cleaned to obtain the cleaning SQL statements, execute each cleaning SQL statement, and record the cleaned data rows.
[0007] Furthermore, the business module and the process module are combined to build a business governance process, which is to combine one or more of the medical business model, data verification configuration, model relationship configuration, and model mapping configuration into one or more of the data collection unit, data verification unit, business verification unit, and data cleaning unit according to different business scenarios and different process requirements.
[0008] The beneficial effects of the present invention are: The solution of the present invention adopts object-oriented thinking, encapsulates and combines Kettle's modules according to medical data governance scenarios, and implements corresponding small modules based on application scenarios. When applying, data governance can be achieved by simply assembling the corresponding application modules; and deeply binds Kettle's processes with medical standard data elements and business models, reducing the dependence of data developers on medical business development capabilities, while also reducing the dependence of medical business developers on data development capabilities, and performing professional diversion, so that people who understand open business can focus on developing business, and people who understand data development can focus on processes. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1It is a flow chart of the packaging method of an embodiment of the present application.
[0010] Figure 2 It is a flowchart of the steps involved in constructing a data model to form a business module in an embodiment of the present application.
[0011] Figure 3 It is a schematic diagram of constructing a medical business model in an embodiment of the present application.
[0012] Figure 4 It is a schematic diagram of the data verification configuration processing of an embodiment of the present application.
[0013] Figure 5 It is a schematic diagram of the model relationship configuration processing of an embodiment of the present application.
[0014] Figure 6 It is a schematic diagram of the model mapping configuration processing of an embodiment of the present application.
[0015] Figure 7 It is the Kettle process of the data acquisition unit in the embodiment of the present application.
[0016] Figure 8 It is the Kettle process of the data verification unit in the embodiment of the present application.
[0017] Fig. 9 It is the Kettle process of the business verification unit in the embodiment of the present application.
[0018] Fig.10 It is the Kettle process of the data cleaning unit in the embodiment of the present application.
[0019] Fig.11 It is a schematic diagram of the combination of the business module and the process module of the embodiment of the present application. DETAILED DESCRIPTION
[0020] To make the objectives, technical solutions and advantages of the embodiments of the present invention more clear, the implementation modes of the present invention are described in detail below with reference to the accompanying drawings. However, the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all the embodiments.
[0021] The present application embodiment provides a packaging method for implementing medical data management based on Kettle, such as Figure 1 As shown, the following process is included: S100, constructing a data model based on health standard data elements to form a business module; S200, build process modules through Kettle; S300, combining the business module and the process module to construct a business governance process; wherein the process module is used to clarify the governance process, and the business module is used to fill in the content to clarify the specific governance content.
[0022] Specifically, Figure 2 As shown, building a data model based on health standard data elements to form a business module includes building a medical business model, data verification configuration processing, model relationship configuration processing, and model relationship configuration processing.
[0023] Among them, building a medical business model includes: inheriting data elements according to reference standards, and combining multiple data elements into a medical business model, such as Figure 3 The figure shows an example, where the data element data type S3, representation format N1, and allowed value CV02.01.102 are used to describe the basic metadata of medical information. The reference standard is one or more of the national standard health information data element, local medical business standard, and hospital business standard. The medical business model is used to generate the table structure of the corresponding database for data storage.
[0024] The data verification configuration processing includes: extracting corresponding verification rules through parsing data elements to form data verification configuration, such as Figure 4 The following is an example where data validation configuration supports being referenced by model fields.
[0025] Model relationship configuration processing includes: based on the constructed medical business model, supplementing the correspondence between models to clarify the business relevance between businesses, such as Figure 5 An example is shown.
[0026] Model mapping configuration processing includes: mapping models with the same business meaning between models according to business requirements to generate model mapping configurations; and supplementing the generated model mapping configurations with the mapping relationships required during mapping conversion for subsequent data cleaning. Figure 6 An example is shown.
[0027] In S200, building process modules through Kettle includes building the following process units separately: Data collection unit: used to read the medical business model and model mapping configuration from the business module to complete data collection from the source table to the target table. The source table and the target table are pre-bound to the medical business model and are consistent with the structure of the bound medical business model.
[0028] Data verification unit: used to read data verification configuration from the business module to complete data verification.
[0029] Business verification unit: used to read the model relationship configuration from the business module to complete the business relationship verification.
[0030] Data cleaning unit: used to read the model relationship configuration from the business module according to the fields and configurations that need to be cleaned to obtain the cleaning SQL statements, execute each cleaning SQL statement, and record the cleaned data rows.
[0031] In S300, the business module and process module are combined to build a business governance process, such as Fig.11 As shown, specifically, according to different business scenarios and different process requirements, one or more of the medical business model, data verification configuration, model relationship configuration, and model mapping configuration are combined into one or more of the data acquisition unit, data verification unit, business verification unit, and data cleaning unit. Fig.11 The example of combination in the application scenario of the outpatient business segment is shown in the figure. According to the individual process modules built by S100 and S200, developers only need to select different businesses and different processes in different business scenarios to build data governance processes. In this diagram, a data collection process can be built between the outpatient business front-end library and the outpatient business governance library based on the data collection unit, and the medical business model is combined in this process to read data; after reaching the outpatient business governance library, a data verification process link can be added based on the data verification unit, and combined with the data verification model; further, from the outpatient business governance library to the outpatient business application library, a data cleaning process can be built based on the data cleaning unit, which is combined with the model mapping configuration; after reaching the outpatient business application library, a business verification process can be built based on the business verification unit, which is combined with the model relationship configuration, thereby completing the orchestration of a business governance process of the outpatient business segment.
[0032] As a further detailed implementation scheme of this example, the Kettle process of the data acquisition unit is as follows Figure 7 As shown, specifically including the following: When the database connection verification is normal, read the medical business model in the business module to obtain the table names of the source table and target table of the data collection process; when the database connection verification is abnormal, end the process; Connect to the data source and verify whether the source table and target table exist in the database based on the table name of the medical business model. If they do not exist, an exception prompt will be displayed and the process will be terminated. If they do exist, the amount of data to be collected will be counted according to the dimension of the data access party. According to the statistical data volume to be collected, the source table data is loaded and processed. During data processing, the model mapping configuration in the business module is read to perform dictionary transcoding, file collection, and primary index conversion. The correct data after data processing is stored in the target table as new data, and the wrong data is recorded and re-collected after updating the data; Statistics of new data, updated data, and error data in each dimension.
[0033] As a further detailed implementation plan of this example, the Kettle process of the data verification unit is as follows: Figure 8 As shown, specifically including the following: When the database connection verification is normal, verify the data volume of the target table being verified; when the database connection verification is abnormal, end the process; If the amount of data in the target table being verified is not 0, the data volume is calculated and the data verification configuration in the business module is read in batches to verify each data. The data in each column of the target table is verified to see whether it matches the type, length, size, value range or regular expression set by the model. If at least one item does not match, the unmatched data is marked as an exception or output to the exception table and the process is terminated. If all match, it is used as the verified data for subsequent processes. If the amount of data in the verification table is 0, the process is terminated.
[0034] As a further detailed implementation plan of this example, the Kettle process of the business verification unit is as follows Fig. 9 As shown, specifically including the following: When the database connection verification is normal, the model relationship configuration is read from the business module to build the association verification condition; when the database connection verification is abnormal, the process ends; Verify the data volume of the target table to be verified. When the verified data volume is not 0, build an associated query SQL script based on the association verification conditions, configure the judgment conditions for the SQL script execution results, and run the SQL script to perform business verification. Record the number of data that passed and failed the verification according to the dimension of the data access party, and mark the data that failed the verification as anomalies or output it to the exception table to end the process. The data that passed the verification is used for subsequent processes. When the verified data volume is 0, end the process.
[0035] As a further detailed implementation plan of this example, the Kettle process of the data cleaning unit is as follows Fig.10 As shown, specifically including the following: When the database connection verification is normal, verify the number of data entries in the target table being cleaned; when the database connection verification is abnormal, end the process; If the number of data is not 0, check whether there are missing values in the data. If not, skip the data and check the next data. If there are missing values, find the associated medical business model through the field information of the missing values, read the corresponding model mapping configuration from the business module, and extract the column values of other medical business models through the mapping relationship of the model mapping configuration to supplement the values. During the value supplement operation, automatically build a cleaning SQL statement based on the mapping relationship, then execute each cleaning SQL statement to clean the data, and record the execution record of the cleaning; If the number of verified data items is 0, the process ends.
[0036] The solution of this example compares the data collection and data governance system tools on site. On the one hand, this solution is applied to the medical industry. Combined with the "Health Standard Data Element", the corresponding medical standard model is quickly built through inheritance standards. Business developers focus on encapsulated "business modules", and Etl development engineers focus on "process module development". Implementers can freely combine the two modules to quickly build medical data governance processes, improve system availability, and implementation efficiency.
[0037] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application.
Claims
1. The encapsulation method for medical data governance based on Kettle is characterized by: include: Build data models based on health standard data elements to form business modules; Build process modules through Kettle; Combine business modules and process modules to build business governance processes; the process module is used to clarify the governance process, and the business module is used to fill in content to clarify specific governance content.
2. The encapsulation method for medical data management based on Kettle according to claim 1 is characterized in that: Building a data model based on health standard data elements to form business modules includes: Constructing a medical business model: inheriting data elements according to reference standards, and combining multiple data elements into a medical business model; wherein the data type, representation format, and allowed value of the data element are used to describe the basic metadata of a medical information, and the reference standard is one or more of the national standard health information data element, local medical business standards, and hospital business standards; the medical business model is used to generate the table structure of the corresponding database for data storage; Data verification configuration processing: By parsing the data elements to extract the corresponding verification rules to form the data verification configuration, the data verification configuration can be referenced by the model fields; Model relationship configuration processing: Based on the constructed medical business model, supplement the correspondence between models to clarify the business relevance between businesses; Model mapping configuration processing: Map models with the same business meaning between models according to business requirements and generate model mapping configuration.
3. The encapsulation method for medical data management based on Kettle according to claim 2 is characterized in that: When processing the model mapping configuration, the generated model mapping configuration is supplemented with the mapping relationship required for mapping conversion for subsequent data cleaning.
4. The encapsulation method for medical data management based on Kettle according to claim 3 is characterized in that: Building a process module through Kettle includes building the following process units separately: Data collection unit: used to read the medical business model and model mapping configuration from the business module to complete data collection from the source table to the target table. The source table and the target table are pre-bound to the medical business model and are consistent with the structure of the bound medical business model respectively; Data verification unit: used to read data verification configuration from the business module to complete data verification; Business verification unit: used to read the model relationship configuration from the business module to complete the business relationship verification; Data cleaning unit: used to read the model relationship configuration from the business module according to the fields and configurations that need to be cleaned to obtain the cleaning SQL statements, execute each cleaning SQL statement, and record the cleaned data rows.
5. The encapsulation method for medical data management based on Kettle according to claim 4 is characterized in that: The Kettle process of the data acquisition unit includes: When the database connection verification is normal, read the medical business model in the business module to obtain the table names of the source table and target table of the data collection process; when the database connection verification is abnormal, end the process; Connect to the data source and verify whether the source table and target table exist in the database according to the table name of the medical business model. If they do not exist, an exception prompt will be displayed and the process will be terminated. If they do exist, the amount of data to be collected will be counted according to the dimension of the data access party. According to the statistical data volume to be collected, the source table data is loaded and processed. During data processing, the model mapping configuration in the business module is read to perform dictionary transcoding, file collection, and primary index conversion. The correct data after data processing is stored in the target table as new data, and the wrong data is recorded and re-collected after updating the data; Statistics of new data, updated data, and error data in each dimension.
6. The packaging method for medical data management based on Kettle according to claim 4 is characterized in that: The Kettle process of the data verification unit includes: When the database connection verification is normal, verify the data volume of the target table being verified; when the database connection verification is abnormal, end the process; If the amount of data in the target table being verified is not 0, the data volume is calculated and the data verification configuration in the business module is read in batches to verify each data. The data in each column of the target table is verified to see whether it matches the type, length, size, value range or regular expression set by the model. If at least one item does not match, the unmatched data is marked as an exception or output to the exception table and the process is terminated. If all match, it is used as the verified data for subsequent processes. If the amount of data in the verification table is 0, the process is terminated.
7. The packaging method for medical data management based on Kettle according to claim 4 is characterized in that: The Kettle process of the business verification unit includes: When the database connection verification is normal, the model relationship configuration is read from the business module to build the association verification condition; when the database connection verification is abnormal, the process ends; Verify the data volume of the target table to be verified. When the verified data volume is not 0, build an associated query SQL script based on the association verification conditions, configure the judgment conditions for the SQL script execution results, and run the SQL script to perform business verification. Record the number of data that passed and failed the verification according to the dimension of the data access party, and mark the data that failed the verification as anomalies or output it to the exception table to end the process. The data that passed the verification is used for subsequent processes. When the verified data volume is 0, end the process.
8. The encapsulation method for medical data management based on Kettle according to claim 4 is characterized in that: The Kettle process of the data cleaning unit includes: When the database connection verification is normal, verify the number of data entries in the target table being cleaned; when the database connection verification is abnormal, end the process; If the number of data is not 0, check whether there are missing values in the data. If there are no missing values, skip the data and check the next data. If there are missing values, find the associated medical business model through the field information of the missing values, and read the corresponding model mapping configuration from the business module. Use the mapping relationship of the model mapping configuration to extract the column values of other medical business models to supplement the values. When supplementing the values, automatically build a cleaning SQL statement based on the mapping relationship, then execute each cleaning SQL statement to clean the data, and record the execution record of the cleaning. If the number of verified data items is 0, the process ends.
9. The encapsulation method for medical data management based on Kettle according to claim 4 is characterized in that: Combining the business module and the process module to build a business governance process is to combine one or more of the medical business model, data verification configuration, model relationship configuration, and model mapping configuration into one or more of the data collection unit, data verification unit, business verification unit, and data cleaning unit according to different business scenarios and different process requirements.