Medical data final consistency synchronization method and device for end-to-end data alignment

Abstract

The invention discloses a medical data final consistency synchronization method and device for end-to-end data alignment, and belongs to the technical field of medical data synchronization, and the method comprises the steps that a coordinator carries out heartbeat detection on participants in the data synchronization process and issues a task script request instruction; the participant submits data acquisition task script information to the coordinator according to the task script request instruction, the coordinator finds out tasks with different read and write data volumes based on the data acquisition task script information and issues data alignment task script information, and the participant performs self-inspection according to the data alignment task script information; the participant pushes a corresponding data alignment task script information request to the coordinator according to a self-checking result, the coordinator screens out corresponding data alignment task script information according to the data alignment task script information request and sends the data alignment task script information to the participant, and the participant executes data alignment. According to the invention, end-to-end data alignment in the medical data synchronization process can be realized, and the accuracy and consistency of the data are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical data synchronization, and in particular relates to a method and device for synchronizing medical data with final consistency with end-to-end data alignment. Background Art

[0002] In the context of the rapid development of medical information technology, the processing and analysis of medical big data has become the key to improving the quality and efficiency of medical services. For companies involved in the field of medical big data, it is necessary to efficiently and accurately collect large amounts of medical data from regional lower-level medical systems (such as HIS, PASS, SAAS systems, etc.) into analysis libraries (such as TiDB, MongoDB, Elasticsearch and other databases and HDFS, Oss, S3 and other file storage) in the cloud center network environment.

[0003] Regional lower-level medical systems generally use data storage media such as Oracle and MySQL to store massive amounts of medical data. These data cover key information such as patients' medical records, examination results, and medication records, and are extremely valuable for subsequent data fusion, data analysis, data quality control, artificial intelligence applications, and medical auxiliary diagnosis applications. However, due to the diversity of data storage media and the complexity of system architecture, collecting these data from the source library of the regional lower-level medical system to the analysis library of the cloud center faces the challenge of ensuring data synchronization consistency during the end-to-end data collection and synchronization process.

[0004] In multiple end-to-end data synchronization collection scenarios, first, the data collection process from the source database in the hospital to the backup database needs to be executed. Data collection is performed through table-to-table or real-time incremental data log change capture (Change Data Capture, CDC) to reduce the risk of locking the source database, thereby preventing interruptions or pauses in the daily use of the lower-level medical system and avoiding affecting the daily outpatient or admission business in the hospital. Then, the data synchronization process is performed from the backup database to multiple analysis databases of the cloud center node data, and finally integrated into the analysis database. Based on the backup database, full and real-time data collection, full and real-time incremental synchronization of data can achieve table-to-table, view-to-table, SQL statement-to-table, with strong plasticity and high scalability, and supports the operation of merging data from multiple single tables into a large wide table (using views or SQL JOIN to collect data to a large wide table in the analysis database TiDB), which provides great convenience for subsequent data processing and application.

[0005] However, in actual operation, you may encounter abnormal situations such as occasional network interruptions, full database connection pool connections, modified source database table fields, and length restrictions on source data table fields during real-time data collection. These abnormalities often lead to end-to-end (source read database and target write database) data inconsistencies and missing data in data synchronization tasks, which seriously affect subsequent data application, processing, and governance work, resulting in the need to reprocess data governance and invest more manpower and workload to investigate, increasing the difficulty and challenges of data collection and greatly reducing the accuracy of data collection.

[0006] At present, end-to-end data synchronization consistency is a common problem faced by companies related to data solutions. In the field of medical big data, the accuracy and consistency of data are directly related to doctors' diagnostic decisions and patients' treatment effects. Therefore, solving the problem of data synchronization consistency and improving the accuracy and stability of data collection are of great significance to promoting the in-depth application of medical big data, and it is necessary to continuously explore and innovate data synchronization solutions. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide a medical data final consistency synchronization method and device with end-to-end data alignment, taking the data synchronization acquisition system as the role of a participant, and creating a coordination system as a coordinator. By having the participants and coordinators execute processes including heartbeat detection, pre-check and data alignment, end-to-end data alignment can be achieved in the medical data synchronization process, thereby ensuring the accuracy and consistency of the data.

[0008] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows: In a first aspect, an embodiment of the present invention provides a method for synchronizing medical data with final consistency with end-to-end data alignment, comprising the following steps: In the heartbeat detection phase, the coordinator is responsible for performing heartbeat detection on all participants in the data synchronization process and issuing task script request instructions; In the pre-check phase, all participants submit data collection task script information to the coordinator according to the task script request instruction. The coordinator finds out the tasks with differences in the amount of read and written data based on the data collection task script information and sends the data alignment task script information. The participants perform self-checks according to the data alignment task script information. In the data alignment stage, participants push the corresponding data alignment task script information request to the coordinator based on the self-check results. The coordinator filters out the corresponding data alignment task script information based on the data alignment task script information request and sends it to the participants. The participants perform data alignment based on the filtered data alignment task script information.

[0009] Preferably, the coordinator is responsible for performing heartbeat detection on all participants in the data synchronization process and issuing task script upload request instructions, including: All participants in the data synchronization process send heartbeat requests to the coordinator. The coordinator receives the heartbeat requests and sends the task script request instructions of the corresponding participants to the topic of the message queue Kafka for the participants to subscribe.

[0010] Preferably, all the participants submit data collection task script information to the coordinator according to the task script request instruction. The coordinator finds out the tasks with differences in the amount of read and write data based on the data collection task script information and issues data alignment task script information, including: After all participants subscribe to the task script request instructions in the topic of the message queue Kafka, they submit the data collection task script information to the coordinator. The coordinator builds a data collection task list based on the data collection task script information. The coordinator starts the scheduled task through the time executor to query the data collection task list, and finds out the tasks in the data collection task list that have differences in the amount of data read from the source library and written to the target library. At the same time, the coordinator sends the data alignment task script information corresponding to the tasks with differences to the corresponding message queue Kafka topic. Participants subscribe to the data alignment task script information under the topic of the message queue Kafka.

[0011] Preferably, the participant performs self-checking according to the data alignment task script information, including: Participants start the self-inspection process for self-inspection based on the obtained data alignment task script information. Through the script information of the read source and write source in the data alignment task script information, the corresponding SQL query statements are executed through the JDBC API interface to obtain the total statistics of the read source data table and the total statistics of the write source data table. If the total statistics of the read source data table and the total statistics of the write source data table are equal, it means that there is no abnormality in the data collection task. If the total statistics of the read source data table and the total statistics of the write source data table are not equal, it means that there is an abnormality in the data collection task and enters the data alignment stage.

[0012] Preferably, the participant pushes the corresponding data alignment task script information request to the coordinator according to the self-check result, and the coordinator screens out the corresponding data alignment task script information according to the data alignment task script information request and sends it to the participant, and the participant performs data alignment according to the screened data alignment task script information, including: Participants send data alignment task script information requests that require data alignment to the coordinator based on their self-check results. The coordinator fills the data alignment tasks into the task queue based on the data alignment task script information requests and filters out the data alignment task script information corresponding to the data alignment tasks one by one by executing SQL statements, and maintains the data alignment task status information in the status table. The coordinator sends the data alignment task script information corresponding to the filtered data alignment tasks to the topic Topic of the message queue Kafka at regular intervals through the time executor. After subscribing to the data alignment task script information corresponding to the data alignment task under the topic Topic of the message queue Kafka, the participants perform the data alignment operation.

[0013] Preferably, when a participant performs a data alignment operation, the following steps are included: Participants create a thread pool. Each thread in the thread pool executes the corresponding data alignment task. The data alignment task corresponds to the thread one by one. The running status variables of the task are managed independently through thread isolation. The data alignment task is executed in parallel by multiple threads to complete the end-to-end data alignment task from the reading source to the writing source.

[0014] Preferably, the corresponding state variable state is modified using the synchronized keyword and the volatile keyword in JAVA to ensure that the atomicity and visibility of the state variable cannot be destroyed in a multi-threaded environment.

[0015] Preferably, all participants execute corresponding threads and read the source table under the source library again through JDBC to obtain the corresponding data records according to the data alignment task script information, and write them in batches to the write source table for storage.

[0016] Preferably, when performing the data alignment operation, the participant further includes: Participants submit the task running status variables of each data collection alignment task to the coordinator, and the coordinator writes the value of the task status variable into the corresponding status table to perform real-time monitoring of the task.

[0017] In a second aspect, an embodiment of the present invention further provides an end-to-end data aligned medical data final consistency synchronization device, which is implemented using the above-mentioned end-to-end data aligned medical data final consistency synchronization method, and includes: a heartbeat detection module, a pre-check module and a data alignment module; The heartbeat detection module is used to perform heartbeat detection on all participants in the data synchronization process through the coordinator and issue task script request instructions; The pre-check module is used to submit data collection task script information to the coordinator through all participants according to the task script request instruction. The coordinator finds out the tasks with differences in the amount of read and written data based on the data collection task script information and issues data alignment task script information. The participants perform self-check according to the data alignment task script information. The data alignment module is used for the participants to push the corresponding data alignment task script information request to the coordinator according to the self-check results. The coordinator filters out the corresponding data alignment task script information according to the data alignment task script information request and sends it to the participants. The participants perform data alignment according to the filtered data alignment task script information.

[0018] Compared with the prior art, the present invention has the following beneficial effects: In a medical system deployed regionally in a LAN environment, where data is stored in the intranet environment of each region, the present invention introduces a coordinator identity system and combines the processes of heartbeat detection, pre-check and data alignment to achieve end-to-end data alignment in the process of medical data synchronization, thereby ensuring the accuracy and consistency of the data. It can efficiently and synchronously collect medical data into a unified analysis library in a cloud center environment, thereby supporting a variety of scenarios such as big data applications, artificial intelligence analysis, and big data-assisted doctors in diagnosis and treatment, showing significant application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a flowchart of a medical data final consistency synchronization method with end-to-end data alignment provided by an embodiment of the present invention; Figure 2 is a schematic diagram of multiple end-to-end data synchronization flow scenarios provided by an embodiment of the present invention; Figure 3 is a schematic diagram of the heartbeat detection stage provided by an embodiment of the present invention; Figure 4 is a schematic diagram of a pre-checking phase and a data alignment phase provided by an embodiment of the present invention; Figure 5 It is a structural diagram of a medical data final consistency synchronization device with end-to-end data alignment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0022] The inventive concept of the present invention is: in view of the problem that end-to-end data synchronization in the prior art is prone to inconsistency, the embodiments of the present invention provide a medical data final consistency synchronization method and device with end-to-end data alignment, through the coordinator taking on the functions of receiving heartbeat requests pushed regularly by all participants, receiving task script information data of all participants, and maintaining task queues, maintaining time executors and maintaining status tables, combined with the two stages of pre-check and data alignment, end-to-end data alignment is achieved in the medical data synchronization process, thereby ensuring the accuracy and consistency of the data.

[0023] In order to ensure the integrity and real-time performance of the data, in the embodiment, Figure 2 In the multiple end-to-end data synchronization flow scenarios shown, a CDC-based data synchronization acquisition system is used to implement data reading and writing from the source database to the backup database in the regional network environment, and data reading and writing from multiple analysis databases to the combined analysis database in the cloud center network environment. A DataWorks-based data synchronization acquisition system or a Datax secondary development XTL tool-based data synchronization acquisition system is used to implement data reading and writing from the backup database to multiple analysis databases. However, in this way, end-to-end data inconsistency is very likely to occur.

[0024] Figure 1 FIG. 1 is a flow chart of a method for synchronizing medical data with final consistency with end-to-end data alignment provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a medical data final consistency synchronization method with end-to-end data alignment, including the following steps: S1, in the heartbeat detection phase, the coordinator is responsible for performing heartbeat detection on all participants in the data synchronization process and issuing task script request instructions.

[0025] In the embodiment, the data synchronization acquisition system is used as a coordinator, and a coordination system is constructed through JAVA application as a coordinator. Figure 3 As shown, the heartbeat detection stage includes: S1.1, all participants in the data synchronization process send heartbeat requests to the coordinator, which is equivalent to reporting, and set the reporting every 5 seconds.

[0026] S1.2, the coordinator receives the heartbeat request and sends the task script request instruction of the corresponding participant to the topic of the message queue Kafka.

[0027] S1.3, all participants subscribe to and consume the task script request instructions in the topic of the message queue Kafka.

[0028] S2, in the pre-check stage, all participants submit data collection task script information to the coordinator according to the task script request instruction. The coordinator finds out the tasks with differences in the amount of read and write data based on the data collection task script information and sends down the data alignment task script information. The participants perform self-checks according to the data alignment task script information.

[0029] S2.1, such as Figure 4 As shown, all participants in data synchronization submit data collection task script information to the coordinator according to the subscribed task script request instruction, and the coordinator builds a data collection task list according to the data collection task script information. The data collection task script information varies according to different participants and is formulated according to specific scenarios. For example, for a data synchronization collection system based on the Datax secondary development XTL tool, the task script information data may include reading source information, library name, table name, connection method, connection information, driver package type, table name, view script, SQL, and writing source information, library name, table name, connection method, connection information, driver package type, write table name, etc. For a data synchronization collection system based on CDC, the task script information data may include reading source information, library name, table name, connection information, connection method, driver package type, incremental field identifier, and writing source information, library name, table name, connection information, connection method, driver package type.

[0030] S2.2, the coordinator starts the scheduled task through the time executor (Timer) to query the data collection task list, and finds out the tasks in the data collection task list where there is a difference in the amount of data read from the source library and written to the target library (the difference is greater than 0), and obtains the data alignment task details, as shown in Table 1. The data alignment task details include reading the source library, reading the source data table name, the source table data volume, the written data volume, the current difference, the write database name, and the write data table name.

[0031] Table 1 Data alignment task details

[0032] S2.3, the coordinator sends the data alignment task script information corresponding to the tasks with differences to the corresponding message queue Kafka topic.

[0033] S2.4, all participants subscribe to the topic of the message queue Kafka to obtain the data alignment task script information.

[0034] S2.5, the participant starts the self-check process according to the obtained data alignment task script information, and executes the corresponding SQL query statement through the JDBC (JAVA Database Connectivity, Java Database Connection) API interface through the script information of the read source and write source in the data alignment task script information. According to the query statistics statement executed, the total number of statistics of the read source data table and the total number of statistics of the write source data table are obtained. If the total number of statistics of the read source data table and the total number of statistics of the write source data table are equal, it means that there is no abnormality in the data collection task. If the total number of statistics of the read source data table and the total number of statistics of the write source data table are not equal, it means that the data collection task has an abnormality and enters the data alignment stage.

[0035] S3, in the data alignment stage, the participant pushes the corresponding data alignment task script information request to the coordinator according to the self-check result. The coordinator filters out the corresponding data alignment task script information according to the data alignment task script information request and sends it to the participant. The participant performs data alignment according to the filtered data alignment task script information.

[0036] S3.1, such as Figure 4 If the self-check result shows that the data collection task is abnormal, it means that data alignment is required. The participant sends a data alignment task script information request corresponding to the data alignment to the coordinator, and the coordinator fills the data alignment task into the task queue according to the data alignment task script information request.

[0037] S3.2, the coordinator executes the corresponding SQL statement according to the data alignment task script information request, filters out the data alignment task script information corresponding to the data alignment task, and maintains the data alignment task status information in the status table. The status table includes the running status of the data alignment task, which includes waiting to be run, running, and completed, and the initialized data alignment task status is filled in as waiting to be run.

[0038] S3.3, the coordinator sends the data alignment task script information corresponding to the data alignment task to the corresponding message queue Kafka topic Topic at regular intervals through the time executor.

[0039] S3.4, all participants perform the data alignment operation after subscribing to the data alignment task script information corresponding to the data alignment task under the Topic of the message queue Kafka.

[0040] S3.5, the participants create a thread pool, each thread in the thread pool executes a corresponding data alignment task, and the data alignment task corresponds to the thread one by one. The running state variables of the task are managed independently through thread isolation, and the corresponding state variables are modified by the keywords synchronize and volatile in JAVA to ensure that the atomicity and visibility of the state variables cannot be destroyed in a multi-threaded environment.

[0041] All participants execute the corresponding threads and read the source table under the source library again through JDBC to obtain the data records of the corresponding time interval according to the data alignment task script information, and write them in batches to the table of the write source for storage. By executing the data alignment task in parallel through multiple threads, the end-to-end data alignment task from the read source to the write source is completed in batches.

[0042] S3.6, all participants submit the task state variable of each data alignment task to the coordinator, and the coordinator writes the value of the task state variable (including running, completed, etc.) into the corresponding state table to monitor the task in real time.

[0043] Based on the same inventive concept, Figure 5 As shown, an embodiment of the present invention further provides a medical data final consistency synchronization device 500 with end-to-end data alignment, including: a heartbeat detection module 510, a pre-check module 520 and a data alignment module 530.

[0044] The heartbeat detection module 510 is used to perform heartbeat detection on all participants in the data synchronization process through the coordinator and issue task script request instructions.

[0045] The pre-check module 520 is used to submit data collection task script information to the coordinator through all participants according to the task script request instruction. The coordinator finds the tasks with differences in the amount of read and write data based on the data collection task script information and issues data alignment task script information. The participants perform self-checks according to the data alignment task script information.

[0046] The data alignment module 530 is used to push the corresponding data alignment task script information request to the coordinator according to the self-check result by the participant. The coordinator filters out the corresponding data alignment task script information according to the data alignment task script information request and sends it to the participant. The participant performs data alignment according to the filtered data alignment task script information.

[0047] It should be noted that the end-to-end data aligned medical data final consistency synchronization device provided in the above embodiment belongs to the same inventive concept as an end-to-end data aligned medical data final consistency synchronization method. The specific implementation process is detailed in an embodiment of an end-to-end data aligned medical data final consistency synchronization method, which will not be repeated here.

[0048] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for synchronizing medical data with eventual consistency with end-to-end data alignment, characterized in that: The following steps are involved: In the heartbeat detection phase, the coordinator is responsible for performing heartbeat detection on all participants in the data synchronization process and issuing task script request instructions; In the pre-check phase, all participants submit data collection task script information to the coordinator according to the task script request instruction. The coordinator finds out the tasks with differences in the amount of read and written data based on the data collection task script information and sends the data alignment task script information. The participants perform self-checks according to the data alignment task script information. In the data alignment stage, participants push the corresponding data alignment task script information request to the coordinator based on the self-check results. The coordinator filters out the corresponding data alignment task script information based on the data alignment task script information request and sends it to the participants. The participants perform data alignment based on the filtered data alignment task script information.

2. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1 is characterized in that: The coordinator is responsible for performing heartbeat detection on all participants in the data synchronization process and issuing task script upload request instructions, including: All participants in the data synchronization process send heartbeat requests to the coordinator. The coordinator receives the heartbeat requests and sends the task script request instructions of the corresponding participants to the topic of the message queue Kafka for the participants to subscribe.

3. The medical data final consistency synchronization method with end-to-end data alignment according to claim 2 is characterized in that: All participants submit data collection task script information to the coordinator according to the task script request instruction. The coordinator finds out the tasks with different read and write data amounts based on the data collection task script information and issues data alignment task script information, including: After all participants subscribe to the task script request instructions in the topic of the message queue Kafka, they submit the data collection task script information to the coordinator. The coordinator builds a data collection task list based on the data collection task script information. The coordinator starts the scheduled task through the time executor to query the data collection task list, and finds out the tasks in the data collection task list that have differences in the amount of data read from the source library and written to the target library. At the same time, the coordinator sends the data alignment task script information corresponding to the tasks with differences to the corresponding message queue Kafka topic. Participants subscribe to the data alignment task script information under the topic of the message queue Kafka.

4. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1, characterized in that: The participant performs self-examination according to the data alignment task script information, including: Participants start the self-inspection process for self-inspection based on the obtained data alignment task script information. Through the script information of the read source and write source in the data alignment task script information, the corresponding SQL query statements are executed through the JDBC API interface to obtain the total statistics of the read source data table and the total statistics of the write source data table. If the total statistics of the read source data table and the total statistics of the write source data table are equal, it means that there is no abnormality in the data collection task. If the total statistics of the read source data table and the total statistics of the write source data table are not equal, it means that there is an abnormality in the data collection task and enters the data alignment stage.

5. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1 or 4, characterized in that: The participant pushes the corresponding data alignment task script information request to the coordinator according to the self-check result. The coordinator screens out the corresponding data alignment task script information according to the data alignment task script information request and sends it to the participant. The participant performs data alignment according to the screened data alignment task script information, including: Participants send data alignment task script information requests that require data alignment to the coordinator based on their self-check results. The coordinator fills the data alignment tasks into the task queue based on the data alignment task script information requests and filters out the data alignment task script information corresponding to the data alignment tasks one by one by executing SQL statements, and maintains the data alignment task status information in the status table. The coordinator sends the data alignment task script information corresponding to the filtered data alignment tasks to the topic Topic of the message queue Kafka at regular intervals through the time executor. After subscribing to the data alignment task script information corresponding to the data alignment task under the topic Topic of the message queue Kafka, the participants perform the data alignment operation.

6. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1, characterized in that: When performing data alignment operations, participants include: Participants create a thread pool. Each thread in the thread pool executes the corresponding data alignment task. The data alignment task corresponds to the thread one by one. The running status variables of the task are managed independently through thread isolation. The data alignment task is executed in parallel by multiple threads to complete the end-to-end data alignment task from the reading source to the writing source.

7. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1, characterized in that: The corresponding state variable state is modified using the JAVA keywords synchronize and volatile to ensure that the atomicity and visibility of the state changes cannot be destroyed in a multi-threaded environment.

8. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1, characterized in that: All participants execute the corresponding threads and read the source table under the source library again through JDBC to obtain the corresponding data records according to the data alignment task script information, and then write them in batches to the write source table for storage.

9. The medical data final consistency synchronization method with end-to-end data alignment according to claim 1, characterized in that: When performing data alignment operations, participants also include: Participants submit the task running status variables of each data collection alignment task to the coordinator, and the coordinator writes the value of the task status variable into the corresponding status table to perform real-time monitoring of the task.

10. An end-to-end data aligned medical data final consistency synchronization device, implemented by the end-to-end data aligned medical data final consistency synchronization method according to any one of claims 1 to 7, characterized in that: Including: heartbeat detection module, pre-check module and data alignment module; The heartbeat detection module is used to perform heartbeat detection on all participants in the data synchronization process through the coordinator and issue task script request instructions; The pre-check module is used to submit data collection task script information to the coordinator through all participants according to the task script request instruction. The coordinator finds out the tasks with differences in the amount of read and written data based on the data collection task script information and issues data alignment task script information. The participants perform self-check according to the data alignment task script information. The data alignment module is used for the participants to push the corresponding data alignment task script information request to the coordinator according to the self-check results. The coordinator filters out the corresponding data alignment task script information according to the data alignment task script information request and sends it to the participants. The participants perform data alignment according to the filtered data alignment task script information.

Images