Rare blood type heterogeneous data synchronization method

Abstract

The invention provides a rare blood type heterogeneous data synchronization method. Rare blood type heterogeneous data can be read from all medical institutions. The method comprises the following steps: preprocessing data, classifying and encrypting the data, transmitting the data to a data center through a medical system special data channel, cleaning rare blood type heterogeneous data, converting the data into a uniform standard format, and transmitting the data to a rare blood type special library for storage after data verification. And synchronization and library falling of rare blood type heterogeneous data of each medical institution are realized. Through the technical scheme, real-time synchronization of rare blood type data in a regional range can be realized, information islands among rare blood types of medical institutions are eliminated, and possibility is provided for information sharing and unified scheduling of rare blood type which is a rare medical resource.

Description

Technical Field

[0001] The present invention relates to the field of medical database construction, and in particular to a rare blood type heterogeneous data synchronization method. Background Art

[0002] Since modern times, with the continuous development of the medical field, people have been constantly updating the construction of medical databases. As an indispensable part of the medical field, blood data plays a significant role in various medical institutions to maintain the normal operation of the medical system. However, the database of my country's medical system is limited by technical reasons in the construction era, and there is also a lack of relevant specifications in the field of blood information. There is no unified blood database standard, which makes blood data distributed in different medical databases. At the same time, the medical databases in various medical institutions also show the characteristics of different database types, different data formats, and differentiated data structures. This makes it impossible for blood data between medical institutions to communicate with each other, forming blood information islands, which not only affects the analysis of blood data and the scheduling of blood resources in the medical system, but also hinders the informatization construction of my country's medical field.

[0003] The existing heterogeneous data synchronization technology in the big data field is mainly used in the migration and maintenance of enterprise data, and rarely pays attention to the data synchronization needs in the medical field. At the same time, because medical data is different from enterprise data, it has the characteristics of high complexity, high privacy, high precision, and high correlation. In the field of medical heterogeneous data synchronization, there is no suitable solution that can meet the heterogeneous data synchronization needs of the medical system. In the blood scenario, there is currently no technical solution to solve the synchronization of rare blood type heterogeneous data. Summary of the invention

[0004] The purpose of the present invention is to solve the technical problems in the above background and to propose a rare blood type heterogeneous data synchronization method, comprising the following steps: S1. Create a log collection user in the medical database of each medical institution, grant various permissions to read log information, and allow Flink CDC to access the medical database of each medical institution through the log collection user; S2. Deploy the Flink computing framework and the Flink CDC data change capture framework built based on Flink in the front-end machines of each medical institution. Use the Connector provided by Flink CDC to read the log streams in each medical database and capture the incremental blood data. S3, preprocess the collected log stream through the DataStream interface provided by the Flink computing framework; S4, classify the preprocessed rare blood type DataStream data stream by identifying specific fields in the rare blood type RowData data; S5. Use the SM4 national encryption algorithm to encrypt the classified rare blood type RowData data, and transmit it to the Kafka queue of the data center for storage through the dedicated data channel of the medical system; S6. By subscribing to the topic partition of the Kafka queue, the back-end Flink cluster of the data center obtains the ciphertext of the rare blood type data stream of each medical institution; after decrypting it using the SM4 decryption algorithm, the DataStream interface of the Flink cluster is used to convert and process the rare blood type data stream; S7. Design a three-level error handling mechanism to handle errors that occur during the conversion of rare blood type DataStream; S8. In the Sink operator of the Flink cluster, perform data verification on the converted rare blood type data, and output the data to the rare blood type storage partition or the error data storage partition in the rare blood type special library according to the result; S9. In combination with blood information management standards and medical database design specifications, a rare blood type database with unified standards will be established in the rare blood type special database to store rare blood type data from different medical institutions.

[0005] In a preferred solution, the medical database in step S1 includes blood transfusion management, HIS, LIS, hand numbness and mobile nursing medical databases; the rare blood type database in step S9 includes a rare blood type storage partition database and an error data storage partition database.

[0006] In a preferred embodiment, the pre-processing in step S3 includes the following steps: S31, deduplication: Use the SHA-256 hash algorithm to generate a custom Transformation deduplication operator, use the deduplication operator to calculate the hash value corresponding to each collected RowData data, and save the corresponding hash value information; According to the specific key fields in the RowData data, the corresponding hash value is generated and compared with the saved hash value. When the hash value is repeated, it means that the blood increment information is repeatedly collected. The blood increment information collected this time should be discarded to avoid duplicate log records caused by log retransmission due to various factors; For RowData data from different medical databases, in combination with actual business scenarios, the corresponding key fields are specified according to the type and characteristics of RowData data, and hash value generation strategies are configured for RowData data from different medical databases; when selecting key fields, fields that are prone to change are excluded; S32. Preliminary screening: Use the Filter rare blood type screening operator to define rare blood type screening rules through Lambda expressions; in the DataStream data stream, identify specific fields related to rare blood types in the RowData data, compare the specific fields in the RowData data with the preset rare blood type feature set, and identify and screen out the RowData data that meets the rare blood type features; S33, preliminary cleaning: Use MapFunction rare blood type preliminary cleaning operator to define preliminary cleaning logic code to achieve the following functions: Remove extra spaces, tabs, and other invalid characters from the rare blood type RowData data; Clean the fields in the rare blood type RowData data, verify whether the format of the key fields meets the preset requirements, and convert the relevant fields into a standard format through the defined conversion logic; For rare blood type RowData data that has errors during the cleaning process, record the error log and perform exception handling. When analyzing the error data later, adjust the cleaning logic in a timely manner in combination with the error log.

[0007] In a preferred solution, the encryption process in step S5 specifically includes the following steps: S51, determining a 128-bit key for encryption and decryption, and grouping the key fields of the rare blood type data to be encrypted into units of 128 bits; and using a padding algorithm to fill in the groups that are less than 128 bits; S52, setting the encryption round number k and the initial state, wherein the initial state setting includes initialization of the key and generation of the round key; S53, executing round function F for each rare blood type data group, wherein the round function F includes nonlinear transformation and linear transformation, wherein: the nonlinear transformation uses an S-box to perform byte replacement on the key field of the input rare blood type data, and the linear transformation transforms the key field of the input rare blood type data by circular left shift; S54, performing several rounds of iterative encryption on each group. In each round, the data of the current group is XORed with the round key and input into the round function F to obtain the data of the next round. The output of the last round is the ciphertext of the key field of the rare blood type data. S55. Use the ciphertext of the key field to replace the original plaintext to form the rare blood type data ciphertext, thereby realizing encryption of the rare blood type data.

[0008] In the preferred solution, in step S6, each element in the rare blood type DataStream, i.e., RowData data, is converted, including the following parallel processing steps: S61. Replace the field name specified in each element with a unified field name; S62, arranging several fields in each element in a specified order; S63. Adjust the specified field in each element to a unified field type and field length.

[0009] In the preferred solution, when different degrees of data errors occur during the processing of the rare blood type RowData data in the Transform operator (a general term for operators used in Flink), the three-level error handling mechanism in step S7 sends the erroneous RowData data to different side output streams for processing in the corresponding operator according to the level of the error. The error levels include: Minor error: RowData data that was not cleaned properly during the initial cleaning phase of preprocessing: Use the primary side output stream to receive RowData data with minor errors, and use the preset logic code to try to repair it. When the repair is successful, perform subsequent conversion processing in the primary side output stream, and output it to the rare blood type special library for execution through the Sink operator of the primary side output stream; when the primary side output stream fails to repair, send the erroneous RowData data to the secondary side output stream to continue trying to repair it; Medium error: For RowData data with missing key fields or mismatched key field types, including: missing collection date, missing blood type label, blood volume should be an integer but is actually a decimal or character, and missing timestamp: Use the secondary side output stream to receive RowData data with medium errors and erroneous RowData data sent from the primary side output stream; by identifying the source of these RowData data, query the complete rare blood type information in the source database to fill or replace the missing or abnormal related fields. When the repair is successful, subsequent conversion processing is performed in the secondary side output stream, and output to the rare blood type special library for execution through the Sink operator of the secondary side output stream; when the secondary side output stream fails to be repaired, the erroneous RowData data is sent to the tertiary side output stream; Serious error: For RowData data whose data source cannot be determined and whose fields have conflict errors: Use the tertiary side output stream to receive the RowData data with serious errors and the erroneous RowData data sent from the secondary side output stream, and directly output the erroneous data to the erroneous data storage partition of the rare blood type special library through the Sink operator of the tertiary side output stream, so as to facilitate subsequent manual troubleshooting of the cause of the error.

[0010] In the preferred solution, in step S8, before the rare blood type data of each medical institution is finally output to the rare blood type special library through the Sink operator, data verification is performed; Through the defined data verification rules, the rare blood type data is verified in the Sink operator: verify whether the rare blood type data conforms to the predetermined format, verify whether the field name and field type of the rare blood type data are correct, and verify the timeliness and update status of the timestamp of the rare blood type data; When data anomalies are detected, a three-level error handling mechanism is used to attempt to repair the data, and the data is output to the rare blood type data storage partition or the error data storage partition of the rare blood type special library through the Sink operator configured in the output stream on different sides.

[0011] In the preferred solution, in the task processing of the entire Flink cluster, the client front end receives the submitted job code, and the JobManager schedules the TaskManager to execute synchronization tasks and manage checkpoints; TaskManager is used for the actual processing of rare blood type data. Each TaskManager runs in an independent JVM process. Task Slot shares the memory resources of TaskManager to transmit and process rare blood type RowData data.

[0012] In the preferred solution, after the Flink cluster uses the DataStream interface to transform and verify the rare blood type RowData data, the required key field values ​​are extracted from the RowData data through the pre-defined logic code in the Sink operator, and passed as parameters to the PreparedStatement interface provided by JDBC, the rare blood type RowData data is dynamically converted into SQL statements, and output to the rare blood type special library for execution, so as to realize the storage of rare blood type data; The PreparedStatement interface provided by JDBC is used to combine the precompiled SQL template and the passed-in parameters to form a complete SQL statement; when a specific field in the rare blood type RowData data is passed in as a parameter, the original placeholder is replaced and combined with the pre-defined SQL template to generate the SQL statement corresponding to the rare blood type RowData data; the precompilation of the SQL template is only executed once and saved in the cache of the database server, and subsequent replacement of parameters does not require recompilation of the SQL statement.

[0013] In the preferred solution, a connection pool is used to manage the connection between the Sink operator and the rare blood type database; The Sink operator accesses the connection pool to obtain a connection that has been established with the rare blood type special database. When the rare blood type data synchronization is completed, the Sink operator returns the connection to the connection pool without destroying the connection with the special database.

[0014] The beneficial effects of the present invention are: (1) Through this technical solution, by utilizing the real-time synchronization feature of Flink CDC, real-time data synchronization from various medical institutions to the rare blood type database is achieved, so that the rare blood type database can quickly respond to data changes and update the rare blood type information in various medical institutions in real time.

[0015] (2) Through the three-level error handling mechanism proposed in this technical solution, it is possible to achieve hierarchical processing of rare blood type error data, use different side output streams to handle errors of corresponding levels, and ensure the processing efficiency and stability of the main data stream. At the same time, through multi-level transmission between side output streams, it is possible to try to repair and store erroneous rare blood type data to the greatest extent possible, and try to ensure the availability of each piece of rare blood type data.

[0016] (3) The SM4 symmetric encryption algorithm used in this technical solution can avoid data leakage that may occur during the transmission process and ensure the security of rare blood type data transmission.

[0017] (4) Through this technical solution, it is possible to realize the collection, preprocessing, transmission, conversion and storage of rare blood type heterogeneous data while ensuring the security, high privacy and high accuracy of such medical data.

[0018] (5) Through this technical solution, the integration of rare blood type resources within the region is realized, the information islands between rare blood types in various medical institutions are eliminated, and the possibility of information sharing and unified scheduling of rare blood types, a scarce medical resource, is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the overall flow chart of the rare blood type heterogeneous data synchronization method.

[0020] Figure 2 This is a simplified flow chart of the rare blood type heterogeneous data synchronization method.

[0021] Figure 3 It is a flowchart of the rare blood type heterogeneous data synchronization method.

[0022] Figure 4 It is a flowchart of the three-level error handling mechanism designed for the rare blood type heterogeneous data synchronization method.

[0023] Figure 5It is a schematic diagram of the structure of the medical database and the rare blood type special database in the rare blood type heterogeneous data synchronization method. DETAILED DESCRIPTION

[0024] This patent proposes Figures 1 to 5 A rare blood type heterogeneous data synchronization solution is shown, which can read rare blood type heterogeneous data from various medical institutions. After preprocessing the data, the data is classified and encrypted, and transmitted to the data center through the dedicated data channel of the medical system. The rare blood type heterogeneous data is cleaned and converted into a unified standard format. After data verification, it is transmitted to the rare blood type special library for storage, so as to realize the synchronization and storage of rare blood type heterogeneous data of various medical institutions. Through this technical solution, real-time synchronization of rare blood type data within the region can be achieved, eliminating information islands between rare blood types in various medical institutions, and providing possibility for information sharing and unified scheduling of rare blood types, a scarce medical resource.

[0025] How it works Create a log collection user in the medical database of each medical institution and grant it various permissions to read log information, so that Flink CDC can access the medical database of each medical institution through the log collection user, including the databases of medical systems such as blood transfusion management, HIS, LIS, hand numbness, and mobile nursing.

[0026] In various medical institutions, blood data is not stored in a unified database, but distributed in different medical databases. The blood transfusion management database mainly stores blood usage data, blood matching data, and blood inventory data in medical institutions. The HIS database stores patient diagnosis-related data, the LIS database stores patient blood test data, the hand numbness database stores surgical blood usage data, and the mobile nursing database contains data on the entire blood transfusion nursing process.

[0027] Therefore, if you want to obtain the log information of blood data from medical institutions, you need to specify the log information related to blood data through the configuration file when reading the logs of various medical databases. In this way, blood data can be extracted from medical databases such as blood transfusion management, HIS, LIS, hand numbness, and mobile nursing, and the blood data log information can be read.

[0028] The Flink computing framework and Flink CDC, a data change capture framework built based on Flink, are deployed in the front-end machines of each medical institution. The Connector provided by Flink CDC is used to read the log streams in each medical database in a targeted manner to achieve incremental capture of blood data.

[0029] For the historical blood data in various medical institutions, a full synchronization strategy is adopted to directly read the relevant storage files containing blood data in the medical database. Through the initialization snapshot function of Flink CDC, different reading strategies are configured according to actual synchronization requirements, such as time point-based reading, specific location-based reading (specified reading of storage files containing blood data), etc., to achieve full capture of historical blood data.

[0030] The connectors provided by Flink CDC will be different for different database types. At the same time, based on different database log formats, Flink CDC will also convert various logs into the RowData format used by Flink's internal DataStream data flow.

[0031] The collected log streams are preprocessed through the DataStream API provided by the Flink computing framework, including deduplication, preliminary screening, and preliminary cleaning using logical code: (1) Deduplication: Use the SHA-256 hash algorithm to generate a custom Transformation deduplication operator. Use the deduplication operator to calculate the hash value corresponding to each collected RowData data and save the corresponding hash value information. The specific steps for generating the hash signature are: based on the specific key fields in the RowData data (excluding fields that are prone to change, such as timestamps), generate the corresponding hash value and compare it with the saved hash value. If the hash value is repeated, it means that the blood increment information is repeatedly collected, and the blood increment information collected this time should be discarded, thereby avoiding duplicate log records caused by log retransmission due to various factors, and realizing log data deduplication in the data preprocessing part.

[0032] For RowData data from different medical databases, the specified key fields will also change. It is necessary to combine the actual business scenarios, specify the corresponding key fields according to the types and characteristics of blood RowData data in different medical databases, and configure the hash value generation strategy for RowData data in different medical databases.

[0033] (2) Preliminary screening: Use the Filter rare blood type screening operator to define rare blood type screening rules through Lambda expressions. In the DataStream data flow, by identifying specific fields related to rare blood types in the RowData data, such as: blood type (Rh negative, AB negative, etc.), blood labels (special labels used to mark rare blood types), antibody types (antibody characteristics related to blood types), clinical diagnosis labels (for example, some patients need specific blood type transfusions), etc., the specific fields in the RowData data are compared with the preset rare blood type feature set to realize the identification and screening of RowData data that meets the rare blood type characteristics in the DataStream data flow.

[0034] (3) Preliminary cleaning: Use the MapFunction rare blood type preliminary cleaning operator to define the preliminary cleaning logic code (using string processing libraries, etc.) to achieve the following functions: Remove extra spaces, tabs, and other invalid characters from the rare blood type RowData data.

[0035] Clean the fields in the rare blood type RowData data and verify whether the format of key fields meets the preset requirements, such as: the blood type field needs to start with a capital letter, the blood label needs to have a specific prefix or suffix, some fields need to meet specific naming specifications, etc., and convert the relevant fields into a standard format through the defined conversion logic.

[0036] For rare blood type RowData data that has errors during the cleaning process, record the error log and perform exception handling. When analyzing the error data later, adjust the cleaning logic in a timely manner in combination with the error log.

[0037] Through the three steps in preprocessing, the collected RowData data can be preliminarily processed. The SHA-256 hash algorithm is used to achieve non-retransmission and non-omission of log streams, thereby improving the reliability of blood data collection. Through the Filter rare blood type screening operator, common blood type information can be distinguished from rare blood type information, thereby realizing the screening and capture of rare blood type RowData data. Through the MapFunction rare blood type preliminary cleaning operator, the preliminary cleaning logic code is configured to remove invalid characters in the DataStream data stream, and at the same time verify and unify the format of key fields in the rare blood type RowData data, ensuring the unification of the rare blood type data format in the same medical institution, thereby reducing the workload of the post-Flink cluster, balancing the workload of the front-end and post-cluster, and improving the overall synchronization efficiency.

[0038] The rare blood type DataStream data stream preprocessed by the Flink computing framework is classified according to the defined logic. Combined with actual business needs, the rare blood type DataStream data stream is classified by identifying specific fields in the rare blood type RowData data, such as: according to the different sources of rare blood type RowData data, the RowData data from the same medical database is classified into one category, and the rare blood type RowData data is classified according to the generation time.

[0039] The classified rare blood type RowData data is encrypted using the SM4 national encryption algorithm. The rare blood type data of the same classification will be transmitted to the same Topic partition of the Kafka queue after encryption. The post-Flink cluster will selectively subscribe to the Topic partition in the Kafka queue, thereby realizing the consumption of rare blood type data from various medical institutions.

[0040] Regarding the encryption of rare blood type data, the SM4 national encryption algorithm mainly encrypts the key fields in the rare blood type RowData data, such as: blood type number, blood type source (storage location), collection time, donor name (ID card, contact information), clinical diagnosis label, blood batch number and other medical sensitive information. The specific encryption steps for performing encryption are as follows: through the pre-defined 128-bit key, the key field that needs to be encrypted in each RowData data is regarded as an encrypted group. Each encrypted group will be iteratively encrypted through the round function and the round key, and the key field will be converted into ciphertext. The ciphertext of the key field is used to replace the original plaintext to achieve the encryption of rare blood type data.

[0041] The encrypted rare blood type data will be transmitted to the Kafka queue for storage through the front-end machine to the medical system dedicated data channel of the rare blood type special library, waiting for decryption and consumption by the back-end Flink cluster. Because the SM4 decryption algorithm has the same structure as the encryption algorithm, but the round keys used are opposite, that is, the decryption round key is the reverse order of the encryption round key. Therefore, the decrypted rare blood type data can be obtained by following the same process in the back-end Flink cluster.

[0042] Within the Kafka queue, multiple partitions can be specified for each Topic to improve parallel processing capabilities and throughput. At the same time, in order to ensure the reliability and fault tolerance of the synchronization process, each partition is also configured with replicas, which are distributed in different Kafka servers (Brokers) to achieve load balancing and fault recovery. Based on the above architecture, for the Kafka queue, the Flink in the front-end machine of each medical institution is the producer, responsible for publishing the encrypted rare blood type data to a specific Topic partition. The back-end Flink cluster in the rare blood type special library acts as a consumer, which subscribes to the specified Topic partition from the Kafka queue to decrypt and consume the rare blood type data from various medical institutions.

[0043] By subscribing to the Topic partition in the Kafka queue, the post-Flink cluster can obtain the rare blood type data in the medical database of each medical institution. The SM4 decryption algorithm is used to decrypt the ciphertext of the rare blood type data stream obtained from the Kafka queue. Because the data format before encryption is the RowData format of the DataStream data stream, the post-Flink cluster can directly process the decrypted rare blood type DataStream data stream.

[0044] Use the Transform operator provided by Flink's DataStream API to implement the corresponding operator interface, define the transformation logic, and process each element (RowData data) in the rare blood type DataStream data stream to achieve the following functions: Replace the specified field name in each element with a unified field name to unify the field names of rare blood type data from different medical institutions.

[0045] Arrange multiple fields in each element in a specified order to unify the order of fields in rare blood type data from different medical institutions.

[0046] Adjust the specified fields in each element to a unified field type and field length, and unify the field types of rare blood type data from different medical institutions.

[0047] In the process of processing the DataStream data stream, each rare blood type RowData data is regarded as an element in the data stream. The element is the basic processing unit of DataStream, and the Flink operator will convert and process each RowData element one by one.

[0048] In the task processing architecture of the Flink cluster, the job code is received by the Job Client, converted into a data stream, and then scheduled by the JobManager. The JobManager is responsible for coordinating the task execution of each TaskManager and managing checkpoints to ensure the consistency and fault tolerance of rare blood type data processing. In a high-availability scenario, multiple JobManagers can be configured, with one as the master and the others as backups, to enhance the reliability of the Flink cluster and ensure stable processing of rare blood type data streams.

[0049] TaskManager is responsible for the actual processing of rare blood type data. Each TaskManager runs in an independent JVM process and has flexible Task Slot configuration to support concurrent execution of data processing tasks. Task Slot shares the memory resources of TaskManager, allowing efficient data transmission and processing, thus ensuring that the Flink cluster can quickly respond to rare blood type data streams from various medical institutions and meet the requirements of high throughput and low latency.

[0050] A three-level error handling mechanism is designed to handle errors that occur during the conversion of rare blood type DataStream data streams through various side output streams marked by OutputTag.

[0051] When the rare blood type RowData data is processed by the Transform operator, data errors of varying degrees occur. According to the level of the error, in the corresponding operator, the erroneous RowData data is sent to different side output streams for processing. The error level is divided into three levels: minor error, medium error, and severe error: (1) Minor errors: mainly for RowData data that was not cleaned properly during the initial cleaning phase of preprocessing, such as: spaces and tabs were not completely removed, key fields were not converted to the required format, uppercase and lowercase did not meet the specifications, etc.

[0052] Use the primary output stream to receive RowData data with minor errors, and use the preset logic code to try to repair it. If the repair is successful, perform subsequent conversion processing in the primary output stream, and output it to the rare blood type special library for execution through the Sink operator of the primary output stream. If the primary output stream fails to be repaired, send the erroneous RowData data to the secondary output stream to continue trying to repair it.

[0053] (2) Medium errors: mainly for RowData data with missing key fields or mismatched key field types, such as missing collection date, missing blood type label, blood volume should be an integer but is actually a decimal or character, missing timestamp, etc.

[0054] Use the secondary side output stream to receive RowData data with medium errors and erroneous RowData data sent from the primary side output stream. By identifying the source of these RowData data, query the complete rare blood type information in the source database to fill in\replace the missing or abnormal related fields. If the repair is successful, subsequent conversion processing is performed in the secondary side output stream, and output to the rare blood type special library for execution through the Sink operator of the secondary side output stream. If the secondary side output stream repair fails, the erroneous RowData data is sent to the tertiary side output stream.

[0055] (3) Serious errors: mainly for RowData data where the data source cannot be determined and field conflicts occur, such as: missing source database, missing blood batch number, conflict between key fields (blood type information is contradictory), etc.

[0056] Use the third-level output stream to receive RowData data with serious errors and erroneous RowData data sent from the second-level output stream. For such errors, it is no longer possible to repair the erroneous RowData data by querying the source database. Directly output the erroneous data to the erroneous data storage partition of the rare blood type database through the Sink operator of the third-level output stream, so that manual troubleshooting of the cause of the error can be performed later.

[0057] Before the rare blood type data of each medical institution is finally output to the rare blood type special library through the Sink operator, data verification is required to ensure the correctness, consistency and integrity of the rare blood type data after conversion. Through the defined data verification rules, the rare blood type data is verified in the Sink operator, such as: verifying whether the rare blood type data conforms to the predetermined format, verifying whether the field name and field type of the rare blood type data are correct, and verifying the timestamp of the rare blood type data to ensure its timeliness and update status. When an error is detected in the data, it will also be sent to the side output stream to try to repair it according to the level of the error, and output to the rare blood type data storage area or error data storage area of ​​the rare blood type special library through the Sink operator configured with different side output streams.

[0058] The three-level error handling mechanism can be used to hierarchically process the erroneous rare blood type RowData data. Through data cleaning of the primary side output stream, source database search of the secondary side output stream, and error storage of the tertiary side output stream, the data consistency, integrity, and accuracy of the rare blood type heterogeneous data during the conversion process are guaranteed. At the same time, the erroneous data is processed separately through the side output stream, which ensures the processing efficiency of the main data stream, reduces the risk of system crash due to erroneous data, enhances the system's fault tolerance, implements a more flexible and refined error handling strategy, and improves the overall stability and reliability of the synchronization process.

[0059] After the Flink cluster uses the DataStream API to transform and verify the rare blood type RowData data, the required key field values ​​are extracted from the RowData data through the pre-defined logic code in the Sink operator and passed as parameters to the PreparedStatement interface provided by JDBC. In this way, the rare blood type RowData data is dynamically converted into SQL statements and output to the rare blood type special library for execution, realizing the unified storage of rare blood type data from different medical institutions.

[0060] The PreparedStatement interface provided by JDBC is used to combine the precompiled SQL template with the passed-in parameters to form a complete SQL statement. When a specific field in the rare blood type RowData data is passed in as a parameter, it will replace the original placeholder and combine with the predefined SQL template to generate the SQL statement corresponding to the rare blood type RowData data. The precompilation of the SQL template is only executed once and saved in the cache of the database server. Subsequent replacement of parameters does not require recompilation of the SQL statement, thereby improving the conversion efficiency of the SQL statement and avoiding the risk of SQL injection. At the same time, it can also prevent the SQL statement from being maliciously tampered with, ensuring the security of the rare blood type data in the database.

[0061] Use the connection pool to manage the connection between the Sink operator and the rare blood type database. The Sink operator can access the connection pool to obtain the established connection with the rare blood type database. When the rare blood type data synchronization is completed, the Sink operator only needs to return the connection to the connection pool without destroying the connection with the special database. The connection pool maintains a certain number of connections to avoid the frequent creation and destruction of connections between the Sink operator and the special database, thereby reducing the connection overhead between the Flink cluster and the rare blood type database, and improving the connection utilization and synchronization efficiency of rare blood type data.

[0062] In combination with blood information management standards and medical database design specifications, a rare blood type database with unified standards is established in the rare blood type special database, which can store rare blood type data in medical databases such as blood transfusion management, HIS, LIS, hand numbness, mobile nursing, etc. in various medical institutions, and realize real-time synchronization of rare blood type data within the region, thereby eliminating information islands between rare blood types in various medical institutions, and providing possibility for information sharing and unified scheduling of rare blood types, a scarce medical resource.

[0063] Through this technical solution, the collection, preprocessing, transmission, conversion and storage of rare blood type heterogeneous data can be achieved while ensuring the security, complexity and high accuracy of such medical data.

[0064] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A rare blood type heterogeneous data synchronization method, characterized by: The following steps are involved: S1. Create a log collection user in the medical database of each medical institution, grant various permissions to read log information, and allow Flink CDC to access the medical database of each medical institution through the log collection user; S2. Deploy the Flink computing framework and the Flink CDC data change capture framework built based on Flink in the front-end machines of each medical institution. Use the Connector provided by Flink CDC to read the log streams in each medical database and capture the incremental blood data. S3, preprocess the collected log stream through the DataStream interface provided by the Flink computing framework; S4, classify the preprocessed rare blood type DataStream data stream by identifying specific fields in the rare blood type RowData data; S5. Use the SM4 national encryption algorithm to encrypt the classified rare blood type RowData data, and transmit it to the Kafka queue of the data center for storage through the dedicated data channel of the medical system; S6. By subscribing to the topic partition of the Kafka queue, the back-end Flink cluster of the data center obtains the ciphertext of the rare blood type data stream of each medical institution; after decrypting it using the SM4 decryption algorithm, the DataStream interface of the Flink cluster is used to convert and process the rare blood type data stream; S7. Design a three-level error handling mechanism to handle errors that occur during the conversion of rare blood type DataStream; S8. In the Sink operator of the Flink cluster, perform data verification on the converted rare blood type data, and output the data to the rare blood type storage partition or the error data storage partition in the rare blood type special library according to the result; S9. In combination with blood information management standards and medical database design specifications, a rare blood type database with unified standards will be established in the rare blood type special database to store rare blood type data from different medical institutions.

2. A rare blood type heterogeneous data synchronization method according to claim 1, characterized in that: The medical database in step S1 includes blood transfusion management, HIS, LIS, hand numbness and mobile nursing medical databases; the rare blood type database in step S9 includes a rare blood type storage partition database and an error data storage partition database.

3. A rare blood type heterogeneous data synchronization method according to claim 1, characterized in that: The preprocessing in step S3 includes the following steps: S31, deduplication: Use the SHA-256 hash algorithm to generate a custom Transformation deduplication operator, use the Transformation deduplication operator to calculate the hash value corresponding to each collected RowData data, and save the corresponding hash value information; According to the specific key fields in the RowData data, the corresponding hash value is generated and compared with the saved hash value. When the hash value is repeated, it means that the blood increment information is repeatedly collected. The blood increment information collected this time should be discarded to avoid duplicate log records caused by log retransmission due to various factors; For RowData data from different medical databases, in combination with actual business scenarios, the corresponding key fields are specified according to the type and characteristics of RowData data, and hash value generation strategies are configured for RowData data from different medical databases; when selecting key fields, fields that are prone to change are excluded; S32. Preliminary screening: Use the Filter rare blood type screening operator to define rare blood type screening rules through Lambda expressions; in the DataStream data stream, identify specific fields related to rare blood types in the RowData data, compare the specific fields in the RowData data with the preset rare blood type feature set, and identify and screen out the RowData data that meets the rare blood type features; S33, preliminary cleaning: Use MapFunction rare blood type preliminary cleaning operator to define preliminary cleaning logic code to achieve the following functions: Remove extra spaces, tabs, and other invalid characters from the rare blood type RowData data; Clean the fields in the rare blood type RowData data, verify whether the format of the key fields meets the preset requirements, and convert the relevant fields into a standard format through the defined conversion logic; For rare blood type RowData data that has errors during the cleaning process, record the error log and perform exception handling. When analyzing the error data later, adjust the cleaning logic in a timely manner in combination with the error log.

4. A rare blood type heterogeneous data synchronization method according to claim 1, characterized in that: The encryption process in step S5 specifically includes the following steps: S51, determining a 128-bit key for encryption and decryption, and grouping the key fields of the rare blood type data to be encrypted into units of 128 bits; and using a padding algorithm to fill in the groups that are less than 128 bits; S52, setting the encryption round number k and the initial state, wherein the initial state setting includes initialization of the key and generation of the round key; S53, executing round function F for each rare blood type data group, wherein the round function F includes nonlinear transformation and linear transformation, wherein: the nonlinear transformation uses an S-box to perform byte replacement on the key field of the input rare blood type data, and the linear transformation transforms the key field of the input rare blood type data by circular left shift; S54, performing several rounds of iterative encryption on each group. In each round, the data of the current group is XORed with the round key and input into the round function F to obtain the data of the next round. The output of the last round is the ciphertext of the key field of the rare blood type data. S55. Use the ciphertext of the key field to replace the original plaintext to form the rare blood type data ciphertext, thereby realizing encryption of the rare blood type data.

5. According to the method for synchronizing rare blood type heterogeneous data as described in claim 1, it is characterized by: step S6 In the process, each element in the rare blood type DataStream, namely the RowData data, is converted, including the following parallel processing steps: S61. Replace the field name specified in each element with a unified field name; S62, arranging several fields in each element in a specified order; S63. Adjust the specified field in each element to a unified field type and field length.

6. A rare blood type heterogeneous data synchronization method according to claim 1, characterized in that: In step S7, when the rare blood type RowData data has different degrees of data errors during the processing of the Transform operator, the three-level error handling mechanism sends the erroneous RowData data to different side output streams for processing in the corresponding operator according to the level of the error. The error levels include: Minor error: RowData data that was not cleaned properly during the initial cleaning phase of preprocessing: Use the primary side output stream to receive RowData data with minor errors, and use the preset logic code to try to repair it. When the repair is successful, perform subsequent conversion processing in the primary side output stream, and output it to the rare blood type special library for execution through the Sink operator of the primary side output stream; when the primary side output stream fails to repair, send the erroneous RowData data to the secondary side output stream to continue trying to repair it; Medium error: For RowData data with missing key fields or mismatched key field types, including: missing collection date, missing blood type label, blood volume should be an integer but is actually a decimal or character, and missing timestamp: Use the secondary side output stream to receive RowData data with medium errors and erroneous RowData data sent from the primary side output stream; by identifying the source of these RowData data, query the complete rare blood type information in the source database to fill or replace the missing or abnormal related fields. When the repair is successful, subsequent conversion processing is performed in the secondary side output stream, and output to the rare blood type special library for execution through the Sink operator of the secondary side output stream; when the secondary side output stream fails to be repaired, the erroneous RowData data is sent to the tertiary side output stream; Serious error: For RowData data whose data source cannot be determined and whose fields have conflict errors: Use the tertiary side output stream to receive the RowData data with serious errors and the erroneous RowData data sent from the secondary side output stream, and directly output the erroneous data to the erroneous data storage partition of the rare blood type special library through the Sink operator of the tertiary side output stream, so as to facilitate subsequent manual troubleshooting of the cause of the error.

7. A rare blood type heterogeneous data synchronization method according to claim 1, characterized in that: In step S8, before the rare blood type data of each medical institution is finally output to the rare blood type special database through the Sink operator, data verification is performed; Through the defined data verification rules, the rare blood type data is verified in the Sink operator: verify whether the rare blood type data conforms to the predetermined format, verify whether the field name and field type of the rare blood type data are correct, and verify the timeliness and update status of the timestamp of the rare blood type data; When data anomalies are detected, a three-level error handling mechanism is used to attempt to repair the data, and the data is output to the rare blood type data storage partition or the error data storage partition of the rare blood type special library through the Sink operator configured in the output stream on different sides.

8. According to claim 1, a rare blood type heterogeneous data synchronization method is characterized in that: in the task processing of the entire Flink cluster, the client front end receives the submitted job code, and the JobManager schedules the TaskManager to execute the synchronization task and manage the checkpoint; TaskManager is used for the actual processing of rare blood type data. Each TaskManager runs in an independent JVM process. Task Slot shares the memory resources of TaskManager to transmit and process rare blood type RowData data.

9. According to claim 1, a rare blood type heterogeneous data synchronization method is characterized by: After the Flink cluster uses the DataStream interface to transform and verify the rare blood type RowData data, it extracts the required key field values ​​from the RowData data through the pre-defined logic code in the Sink operator and passes them as parameters to the PreparedStatement interface provided by JDBC. The rare blood type RowData data is dynamically converted into SQL statements and output to the rare blood type special database for execution, thus realizing the storage of rare blood type data. The PreparedStatement interface provided by JDBC is used to combine the precompiled SQL template and the passed-in parameters to form a complete SQL statement; when a specific field in the rare blood type RowData data is passed in as a parameter, the original placeholder is replaced and combined with the pre-defined SQL template to generate the SQL statement corresponding to the rare blood type RowData data; the precompilation of the SQL template is only executed once and saved in the cache of the database server, and subsequent replacement of parameters does not require recompilation of the SQL statement.

10. A rare blood type heterogeneous data synchronization method according to claim 1, characterized in that: Use the connection pool to manage the connection between the Sink operator and the rare blood type database; The Sink operator accesses the connection pool to obtain a connection that has been established with the rare blood type special database. When the rare blood type data synchronization is completed, the Sink operator returns the connection to the connection pool without destroying the connection with the special database.

Images