Patient primary index construction method and system based on Apache Spark, terminal and medium

Abstract

The invention relates to the technical field of primary indexes, in particular to an Apache Spark-based patient primary index construction method and system, a terminal and a medium, and the method comprises the following steps: configuring a matching factor and a legality verification rule; different matching factors are selected to be combined to configure a plurality of patient information merging rules; automatic merging rules in the patient information merging rules are cyclically traversed in the source patient data result set, and legal source patient data are associated and compared with the main index database to obtain matched patient information and unmatched patient information; the matched patient information result set and the unmatched patient information result set are combined and then stored in a database; and traversing the patient information merging rule in the database, grouping according to the EMP ID of the patient data in the merged result set, and writing the EMP ID of the data with the most complete data quality into the main index table. According to the method and the device, the effects of configurable functions and high main index construction efficiency are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of primary indexes, and in particular to a method, system, terminal and medium for constructing a primary index for patients based on Apache Spark. Background Art

[0002] In the wave of digital transformation, the medical industry is undergoing unprecedented changes, and the application of information technology has greatly improved the quality and efficiency of medical services. However, with the deepening of informatization, the differences and complexity between the information systems provided by hospitals and different suppliers have led to the fragmentation and inconsistency of patient data, which has seriously affected the patient experience and the integration of scientific research data. In addition, in the same information system, due to factors such as repeated file creation, failure to collect ID cards, and the use of other people's medical insurance cards, there will be multiple duplicate records for the same patient, resulting in more dispersed patient data.

[0003] In existing solutions, the method of building a medical patient master index is usually based on data analysis model technology, dynamic multi-threading and cache mechanism technology, or database replication technology. Among them, the technical principle based on data analysis model is to establish a data analysis model by analyzing a large amount of patient data, such as classification algorithms and clustering algorithms in machine learning, and use these models to automatically learn and identify which patient information belongs to the same individual, and then build a master index. The technical principle based on dynamic multi-threading and cache mechanism is to configure patient factors, the weights of each patient factor, and the weight threshold range, create a cache pool for storing different factor information, read data from the patient information database and cache it, and use dynamic multi-threading to build the master index calculation for data that can be processed concurrently. The technical principle based on database replication is to achieve data sharing and information integration by copying data from one database to another.

[0004] However, the above-mentioned method for constructing the main index of medical patients has the problems of low efficiency in processing large-scale data and poor functional scalability. Summary of the invention

[0005] In order to solve the technical problems of low efficiency in processing large-scale data and poor functional scalability in the existing medical patient master index construction method, the present invention provides a patient master index construction method based on Apache Spark, and also provides a patient master index construction system based on Apache Spark, a terminal and a medium.

[0006] To achieve the above objectives, in a first aspect, a technical solution adopted by a method for constructing a patient primary index based on Apache Spark in the present invention is: A method for constructing a patient primary index based on Apache Spark includes the following steps: S1. Configure matching factors and legality verification rules; S2. Select different matching factors to combine and configure a plurality of patient information merging rules, where the patient information merging rules include automatic merging rules and manual merging rules; S3, obtaining the source patient information table, and configuring the correspondence between each field in the source patient information table and each field in the matching factor; S4. Read the source patient data in the source patient information table, generate a source patient data result set, expand the column fields of the source patient data result set, and add a Yes flag to each column field based on the legality verification rule to verify the legality of the current column field attribute; S5. Loop through the automatic merge rules in the patient information merge rules in the source patient data result set, associate and compare the legitimate source patient data with the main index library, obtain the matched patient information and the unmatched patient information, use the original EMP ID for the matched patient information, randomly generate a unique EMP ID for each unmatched patient information, generate a matched patient information result set based on the matched patient information, generate an unmatched patient information result set based on the unmatched patient information, merge the matched patient information result set and the unmatched patient information result set and store them in the database; S6, traversing the database in order according to the order of the patient information merging rules, executing step S7 when encountering an automatic merging rule, and executing step S8 when encountering a manual merging rule; S7, perform group by according to the merging condition, assemble SQL statements and poll the database, select the EMP ID of the data with the earliest generation time among the duplicate data, and update the EMP ID of other data to the EMPID of the data with the earliest generation time, until no duplicate data is found, determine whether the current information merging rule is the last information merging rule, if so, execute S9, if not, return to step S6; S8, stop the current manual merging rule, wait for the staff to perform the merging operation, and determine whether the current information merging rule is the last information merging rule. If so, execute S9, if not, return to step S6; S9. Select the EMP ID of the data with the most complete data quality and write it into the main index table.

[0007] As a preferred implementation of the patient primary index construction method based on Apache Spark in the present application, steps S4-S9 are executed by a Spark cluster based on Apache Spark.

[0008] As a preferred implementation of the patient primary index construction method based on Apache Spark in the present application, the matching factors include some or all of the patient's name, ID number, telephone number, date of birth, gender, marital status, and home address.

[0009] As a preferred implementation of the patient primary index construction method based on Apache Spark of the present application, selecting the EMP ID of the data with the most complete data quality includes: reading the result set after completing all patient information merging rules, grouping the patient data in the result set according to the EMP ID, and evaluating the data quality in the same group of EMP IDs based on whether each attribute has a value.

[0010] In the second aspect, the technical solution adopted by a patient master index construction system based on Apache Spark in the present invention is: A patient master index construction system based on Apache Spark includes a configuration module, a legality verification module, a matching module, a merging module, and a master index construction module, wherein: The configuration module is configured to configure matching factors and legality verification rules; select different matching factors for combination to configure a number of patient information merging rules, the categories of patient information merging rules include automatic merging rules and manual merging rules; obtain the source patient information table, and configure the comparison relationship between each field in the source patient information table and each field in the matching factor; The legitimacy verification module is configured to read the source patient data in the source patient information table, generate a source patient data result set, expand the column fields of the source patient data result set, and add a Yes flag bit to each column field based on the legitimacy verification rule to verify the legitimacy of the current column field attribute; The matching module is configured to loop through the automatic merging rules in the patient information merging rules in the source patient data result set, associate and compare the legitimate source patient data with the main index library, obtain matched patient information and unmatched patient information, use the original EMP ID for the matched patient information, randomly generate a unique EMP ID for each unmatched patient information, generate a matched patient information result set based on the matched patient information, generate an unmatched patient information result set based on the unmatched patient information, merge the matched patient information result set and the unmatched patient information result set and store them in the database; The merge module is configured to traverse the patient information merge rules in the database, execute the automatic merge rules when encountering the automatic merge rules, and execute the manual merge rules when encountering the manual merge rules; executing the automatic merge rules includes performing group by according to the merge conditions, assembling SQL statements and polling the database, selecting the EMP ID of the data with the earliest generation time for the duplicate data, updating the EMP ID of other data to the EMP ID of the data with the earliest generation time, until no duplicate data is found, and executing the next patient information merge rule; executing the manual merge rule includes terminating the current manual merge rule, and executing the next patient information merge rule after the staff performs the merge operation; The primary index building module is configured to, after all patient information merging rules are completed, read the result set after completing all patient information merging rules, group the patient data in the result set according to the EMP ID, evaluate the data quality in the same group of EMP IDs based on whether each attribute has a value, select the EMP ID of the data with the most complete data quality, and write it into the primary index table.

[0011] As a preferred implementation of a patient master index construction system based on Apache Spark, a legitimacy verification module, a matching module, a merging module, and a master index construction module are run on a Spark cluster.

[0012] As a preferred implementation of a patient primary index construction system based on Apache Spark, the primary index construction module selects the EMP ID of the data with the most complete data quality, including: reading the result set after completing all patient information merging rules, grouping the patient data in the result set according to the EMP ID, and evaluating the data quality in the same group of EMP IDs based on whether each attribute has a value.

[0013] In a third aspect, a technical solution adopted by a terminal in the present invention is: A terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned methods for constructing a primary patient index based on Apache Spark are implemented.

[0014] In a fourth aspect, a technical solution adopted by a medium in the present invention is: A storage medium stores a computer program thereon, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for constructing a patient primary index based on Apache Spark are implemented.

[0015] The beneficial effects of the present invention include: This application solves the problem of scattered and inconsistent patient data in the medical field. It adopts the (B / S) architecture, integrates the Spark engine and state process control technology, realizes the configurable patient information merging rules and legality verification rules, and the functions of automatic and manual merging, and builds a unified, efficient and scalable patient information management platform for hospitals, helping the medical industry to move towards a higher level of digital transformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 It is a flowchart of a method for constructing a patient primary index based on Apache Spark in a specific implementation manner of the present invention; Figure 2 This is a structural diagram of a patient primary index construction system based on Apache Spark in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0018] Before describing the technical content of the specific embodiments of the present invention, the professional knowledge involved in the present invention is first introduced: 1.EMP ID is an ID used to identify all similar and identical patients.

[0019] 2. Main index library: contains the basic information of all patients in previous tasks, which is identified by a unique data ID. If the unique data ID of a patient is the same in multiple tasks, only one record will be stored in the library. That is, if a single task is executed multiple times, there will not be multiple identical data records in the main index library.

[0020] 3. Patient master index: In the hospital's information management system, the patient master index is a key index used to uniquely identify each patient. It may be built based on the patient's ID number (ideally, the ID number is unique). For example, when a doctor needs to view all the medical records of a patient, including outpatient medical records, inpatient medical records, examination reports, treatment plans, etc., the system will find all the records corresponding to the patient through the patient master index (with the ID number as the key identifier). These records may be stored in different database tables or files, such as outpatient databases, inpatient databases, etc., but they can be integrated together through the master index to provide medical staff with a complete view of patient information.

[0021] 4. Matching factor: Matching factor is used to determine whether different records belong to the same object (such as the same patient, the same customer, etc.). It is usually some key information, such as name, date of birth, ID number, etc. Matching factor is a key element in building the primary index. By comparing and judging the matching factor, it is determined which records belong to the same object, and these records are associated with the primary index; 5.Apache Spark: Apache Spark is an open source, fast, general, and scalable big data processing framework based on in-memory computing. It provides a variety of data processing capabilities including batch processing, stream processing, machine learning, graph computing, etc. It supports multiple programming languages ​​​​such as Scala, Java, Python, R, etc., and can efficiently process massive data on large-scale clusters.

[0022] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in this specific embodiment. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] Reference Figure 1 , a method for constructing a patient primary index based on Apache Spark proposed in this embodiment includes the following steps: S1. Configure matching factors and legitimacy verification rules. Matching factors include some or all of the patient's name, ID number, phone number, date of birth, gender, marital status, and home address. S2. Select different matching factors to combine and configure a plurality of patient information merging rules, where the patient information merging rules include automatic merging rules and manual merging rules; S3, obtaining the source patient information table, and configuring the correspondence between each field in the source patient information table and each field in the matching factor; S4. Read the source patient data in the source patient information table, generate a source patient data result set, expand the column fields of the source patient data result set, and add a Yes flag to each column field based on the legality verification rule to verify the legality of the current column field attribute; S5. Loop through the automatic merge rules in the patient information merge rules in the source patient data result set, associate and compare the legitimate source patient data with the main index library, obtain the matched patient information and the unmatched patient information, use the original EMP ID for the matched patient information, the original EMP ID is stored in the main index library, randomly generate a unique EMP ID for each unmatched patient information, generate a matched patient information result set based on the matched patient information, generate an unmatched patient information result set based on the unmatched patient information, merge the matched patient information result set and the unmatched patient information result set and store them in the database; S6, traversing the database in order according to the order of the patient information merging rules, executing step S7 when encountering an automatic merging rule, and executing step S8 when encountering a manual merging rule; S7, perform group by according to the merging condition, assemble SQL statements and poll the database, select the EMP ID of the data with the earliest generation time among the duplicate data, and update the EMP ID of other data to the EMPID of the data with the earliest generation time, until no duplicate data is found, determine whether the current information merging rule is the last information merging rule, if yes, execute S9, if no, return to step S6; S8, suspend the current manual merging rule, wait for the staff to perform the merging operation, and determine whether the current information merging rule is the last information merging rule. If yes, execute S9, if not, return to step S6; S9. Read the result set after completing all patient information merging rules, group the patient data in the result set according to the EMPID, evaluate the data quality in the same group of EMP IDs based on whether each attribute has a value, select the EMP ID of the data with the most complete data quality, and write it into the main index table.

[0024] In this embodiment, the EMP ID is used as the primary index to associate the patient's information in different systems. Due to the uniqueness of the EMP ID, it can prevent duplicate records or wrong associations in the database.

[0025] In this embodiment, steps S4-S9 are executed by a Spark cluster based on Apache Spark.

[0026] like Figure 2As shown below, a patient primary index construction system based on Apache Spark is provided in an embodiment of the present disclosure. The patient primary index construction system based on Apache Spark and the patient primary index construction method based on Apache Spark in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the patient primary index construction system based on Apache Spark, please refer to the embodiment of the above-mentioned patient primary index construction method based on Apache Spark.

[0027] A patient master index construction system based on Apache Spark includes a configuration module, a legality verification module, a matching module, a merging module, and a master index construction module, wherein: The configuration module is configured to configure matching factors and legality verification rules; select different matching factors for combination to configure a number of patient information merging rules, the categories of patient information merging rules include automatic merging rules and manual merging rules; obtain the source patient information table, and configure the comparison relationship between each field in the source patient information table and each field in the matching factor; The legitimacy verification module is configured to read the source patient data in the source patient information table, generate a source patient data result set, expand the column fields of the source patient data result set, and add a Yes flag bit to each column field based on the legitimacy verification rule to verify the legitimacy of the current column field attribute; The matching module is configured to loop through the automatic merging rules in the patient information merging rules in the source patient data result set, associate and compare the legitimate source patient data with the main index library, obtain matched patient information and unmatched patient information, use the original EMP ID for the matched patient information, randomly generate a unique EMP ID for each unmatched patient information, generate a matched patient information result set based on the matched patient information, generate an unmatched patient information result set based on the unmatched patient information, merge the matched patient information result set and the unmatched patient information result set and store them in the database; The merge module is configured to traverse the patient information merge rules in the database, execute the automatic merge rules when encountering the automatic merge rules, and execute the manual merge rules when encountering the manual merge rules; executing the automatic merge rules includes performing group by according to the merge conditions, assembling SQL statements and polling the database, selecting the EMP ID of the data with the earliest generation time for the duplicate data, updating the EMP ID of other data to the EMP ID of the data with the earliest generation time, until no duplicate data is found, and executing the next patient information merge rule; executing the manual merge rule includes terminating the current manual merge rule, and executing the next patient information merge rule after the staff performs the merge operation; The primary index building module is configured to, after all patient information merging rules are completed, read the result set after completing all patient information merging rules, group the patient data in the result set according to the EMP ID, evaluate the data quality in the same group of EMP IDs based on whether each attribute has a value, select the EMP ID of the data with the most complete data quality, and write it into the primary index table.

[0028] In this embodiment, the legality verification module, matching module, merging module, and primary index building module run on a Spark cluster.

[0029] An embodiment of the present application also proposes a terminal, comprising a memory, a processor, a communication unit, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a method for constructing a patient primary index based on Apache Spark when executing the program; the memory, the processor, and the communication unit communicate via one or more buses.

[0030] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0031] The processor can be the nerve center and command center of the terminal. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0032] The memory is used to store the execution instructions of the processor, and the memory can be implemented by any type of volatile or non-volatile storage terminal or them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory are executed by the processor, the terminal can execute some or all steps in the above-mentioned embodiment of the method for constructing the patient master index based on Apache Spark.

[0033] The wireless communication function of an electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0034] The wireless communication module can provide wireless communication solutions for electronic devices including wireless LAN, Bluetooth, global navigation satellite system, frequency modulation, short-range wireless communication technology, infrared technology, etc.

[0035] This embodiment further proposes a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods for constructing a patient primary index based on Apache Spark as described above are implemented.

[0036] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a patient primary index based on Apache Spark, characterized in that: The following steps are involved: S1. Configure matching factors and legality verification rules; S2. Select different matching factors to combine and configure a plurality of patient information merging rules, where the patient information merging rules include automatic merging rules and manual merging rules; S3, obtaining the source patient information table, and configuring the correspondence between each field in the source patient information table and each field in the matching factor; S4. Read the source patient data in the source patient information table, generate a source patient data result set, expand the column fields of the source patient data result set, and add a Yes flag to each column field based on the legality verification rule to verify the legality of the current column field attribute; S5. Loop through the automatic merge rules in the patient information merge rules in the source patient data result set, associate and compare the legitimate source patient data with the main index library, obtain the matched patient information and the unmatched patient information, use the original EMP ID for the matched patient information, randomly generate a unique EMP ID for each unmatched patient information, generate a matched patient information result set based on the matched patient information, generate an unmatched patient information result set based on the unmatched patient information, merge the matched patient information result set and the unmatched patient information result set and store them in the database; S6, traversing the database in order according to the order of the patient information merging rules, executing step S7 when encountering an automatic merging rule, and executing step S8 when encountering a manual merging rule; S7, perform group by according to the merging condition, assemble SQL statements and poll the database, select the EMP ID of the data with the earliest generation time among the duplicate data, update the EMP ID of other data to the EMP ID of the data with the earliest generation time, until no duplicate data is found, determine whether the current information merging rule is the last information merging rule, if so, execute S9, if not, return to step S6; S8, stop the current manual merging rule, wait for the staff to perform the merging operation, and determine whether the current information merging rule is the last information merging rule. If so, execute S9, if not, return to step S6; S9. Select the EMP ID of the data with the most complete data quality and write it into the main index table.

2. The method for constructing a patient primary index based on Apache Spark according to claim 1, characterized in that: Steps S4-S9 are executed by a Spark cluster based on Apache Spark.

3. The method for constructing a patient primary index based on Apache Spark according to claim 1, characterized in that: Matching factors include some or all of the patient's name, ID number, phone number, date of birth, gender, marital status, and home address.

4. The method for constructing a patient primary index based on Apache Spark according to claim 1, characterized in that: Selecting the EMP ID of the data with the most complete data quality includes: reading the result set after completing the merging rules of all patient information, grouping the patient data in the result set according to the EMP ID, and evaluating the data quality in the same group of EMP IDs based on whether each attribute has a value.

5. A patient master index construction system based on Apache Spark, characterized in that: It includes configuration module, legality verification module, matching module, merging module and main index building module, among which: The configuration module is configured to configure matching factors and legality verification rules; select different matching factors for combination to configure a number of patient information merging rules, the categories of patient information merging rules include automatic merging rules and manual merging rules; obtain the source patient information table, and configure the comparison relationship between each field in the source patient information table and each field in the matching factor; The legitimacy verification module is configured to read the source patient data in the source patient information table, generate a source patient data result set, expand the column fields of the source patient data result set, and add a Yes flag bit to each column field based on the legitimacy verification rule to verify the legitimacy of the current column field attribute; The matching module is configured to loop through the automatic merging rules in the patient information merging rules in the source patient data result set, associate and compare the legitimate source patient data with the main index library, obtain matched patient information and unmatched patient information, use the original EMP ID for the matched patient information, randomly generate a unique EMP ID for each unmatched patient information, generate a matched patient information result set based on the matched patient information, generate an unmatched patient information result set based on the unmatched patient information, merge the matched patient information result set and the unmatched patient information result set and store them in the database; The merge module is configured to traverse the patient information merge rules in the database, execute the automatic merge rules when encountering the automatic merge rules, and execute the manual merge rules when encountering the manual merge rules; executing the automatic merge rules includes performing group by according to the merge conditions, assembling SQL statements and polling the database, selecting the EMP ID of the data with the earliest generation time for the duplicate data, updating the EMP ID of other data to the EMP ID of the data with the earliest generation time, until no duplicate data is found, and executing the next patient information merge rule; executing the manual merge rule includes terminating the current manual merge rule, and executing the next patient information merge rule after the staff performs the merge operation; The primary index building module is configured to, after all patient information merging rules are completed, read the result set after completing all patient information merging rules, group the patient data in the result set according to the EMP ID, evaluate the data quality in the same group of EMP IDs based on whether each attribute has a value, select the EMP ID of the data with the most complete data quality, and write it into the primary index table.

6. The patient master index construction system based on Apache Spark according to claim 5, characterized in that: The validity check module, matching module, merging module, and main index building module run on the Spark cluster.

7. The patient master index construction system based on Apache Spark according to claim 5, characterized in that: Matching factors include some or all of the patient's name, ID number, phone number, date of birth, gender, marital status, and home address.

8. The patient master index construction system based on Apache Spark according to claim 5, characterized in that: Selecting the EMP ID of the data with the most complete data quality includes: the main index construction module reads the result set after completing the merging rules of all patient information, groups the patient data in the result set according to the EMP ID, and evaluates the data quality in the same group of EMP IDs based on whether each attribute has a value.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a method for constructing a patient primary index based on Apache Spark according to any one of claims 1 to 4 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for constructing a patient primary index based on Apache Spark as described in any one of claims 1 to 4 are implemented.

Images