Medical equipment Internet of Things streaming data management method based on NiFi
Through the NiFi-based IoT data management method for medical equipment, the data island problem caused by the complex access to IoT data of medical equipment is solved, seamless integration and sharing of data is realized, and the quality of medical services is improved.
Patent Information
- Application Number
- CN202411759726.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-14
AI Technical Summary
The complex access to IoT data of medical equipment has led to the widespread existence of data silos, affecting the efficiency of data integration, sharing and use.
Using NiFi-based IoT streaming data management method for medical equipment, streaming data management is realized through NiFi unit, Kafka unit, Druid unit, FlinkSQL unit and MiNiFi unit to achieve seamless integration and sharing of data.
It realizes efficient and flexible access and management of IoT data of medical equipment, eliminates data silos, improves data availability, integration and sharing, and thus improves the quality of medical services.
Smart Images
Figure CN119626428A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to Internet of Things technology, and in particular to a method for managing medical device Internet of Things stream data based on NiFi. Background Art
[0002] At present, data access of the Internet of Things for medical devices faces complex requirements such as cross-hospital, multi-department, multi-type equipment and multi-data structure. These factors make traditional solutions incapable of uniformly processing data from various sources, resulting in the widespread phenomenon of data islands. In this case, when researchers try to use these valuable data, they often need to collect them from each data source one by one, which not only consumes a lot of time and human resources, but also increases the difficulty of data integration. In addition, the lack of uniformity in data formats and structures between different devices and systems further reduces the efficiency and operability of data use. This cumbersome process not only affects research progress, but also restricts the timeliness and scientificity of clinical decision-making.
[0003] In the current medical system, data from various hospitals are collected independently, with inconsistent standards and different collection methods. This fragmented data set feature not only reduces researchers' willingness to use data, but also makes it difficult to convert the collected data into actual value.
[0004] The shortcomings pointed out include: (1) Dispersibility Each hospital and department has established its own independent data collection system, often investing resources repeatedly without knowing each other, resulting in widespread duplication of construction. The lack of communication and collaboration between different units eventually formed data islands. This not only means that the data collected by each party can only be used in their own environment, but also greatly reduces the efficiency of the use of scientific research funds and causes a waste of resources.
[0005] (2) Diversity Since hospitals use different standards in data collection, it is extremely difficult to integrate these data. Differences in data format and structure require tedious data cleaning and conversion, which further increases the cost and complexity of data use. This diversity not only affects the comparability of data, but also limits the possibility of cross-hospital collaboration and research.
[0006] (3) Offline Each hospital collects data independently. Although real-time access is possible within the hospital, real-time sharing is not possible in other hospitals. This offline state limits information exchange and hinders collaboration between researchers in different hospitals, slowing down research progress and decision-making. Therefore, it is urgent to establish a unified, interconnected data management platform to improve data availability, integration, and sharing, thereby promoting scientific research progress and clinical applications.
[0007] (4) Patient information cannot be filled in In the process of medical device IoT, a problem was discovered: the acquired data could not be matched with the patient. This is a very important problem. If the acquired data does not contain patient information, the data cannot be analyzed and applied later, because the data needs to be analyzed with the patient's medical history and personal information. In fact, the patient information can be manually filled in on the medical device, but in actual use, Chinese medical staff basically do not fill it in manually. Summary of the invention
[0008] In view of the above problems, this application aims to propose a medical device IoT stream data management method based on NiFi.
[0009] This application proposes a medical device IoT stream data management method based on NiFi, which implements stream data management through NiFi units, Kafka units, Druid units, FlinkSQL units, and MiNiFi units; The NiFi unit includes a cloud-side NiFi unit and an imaging device-side NiFi unit. The cloud-side NiFi unit is deployed on the cloud server and is responsible for overall management of data flows, including an ICU data parser and an imaging device data parser. The FlinkSQL unit calculates HIS data in real time and generates patient information for real-time query by the cloud-based NiFi unit; The MiNiFi unit is arranged on a sensing terminal connected to the ICU device, processes the data of the ICU device to obtain processed ICU device data, and the processed ICU device data is sent to the cloud NiFi unit; The NiFi unit on the imaging device side is deployed on the edge server connected to the imaging device, and sends the data of the imaging device to the NiFi unit on the cloud side; The ICU data parser of the cloud NiFi unit processes the processed data from the MiNiFi unit, matches it with the patient information obtained by the real-time query, and obtains the matched ICU equipment data; The imaging device data parser of the cloud-based NiFi unit processes the imaging device data from the NiFi unit on the imaging device side, matches it with the patient information obtained by the real-time query, and obtains the matched imaging device data; The matched ICU device data and the matched imaging device data are pushed to the Kafka unit for the Druid unit to use to ingest data from the Kafka unit in real time; The Druid unit is used to realize real-time display and aggregate calculation of data, as well as to provide external data access services.
[0010] Preferably, the MiNiFi unit on the sensing terminal uses the NiFi processor developed by itself to aggregate and push the collected and preliminarily processed ICU equipment data to the cloud-based NiFi unit.
[0011] Preferably, the MiNiFi unit on the sensing terminal communicates with the cloud NiFi unit using HL7 messages.
[0012] Preferably, when parsing the ICU equipment data in the form of HL7 messages, the ICU data parser of the cloud-based NiFi unit will fill in the parsed data in real time according to the corresponding patient information including the bed number obtained by the aggregation calculation of the FlinkSQL unit.
[0013] Preferably, after data cleaning and conversion, the NiFi unit on the imaging device side transmits the imaging device data to the NiFi unit on the cloud side via HTTP.
[0014] Preferably, the cloud-based NiFi unit includes a log file parser; the log file parser monitors the log files in the FTP shared directory set by the imaging device end, obtains and parses the new log file when a new log file is found, and then pushes the parsed data into the Kafka unit.
[0015] Preferably, the ICU device data from the MiNiFi unit includes the indicators to be collected, the data generation timestamp, and the unique identifier corresponding to the ICU device and its sensing terminal.
[0016] Preferably, when the cloud-side NiFi unit processes the received ICU device data, the processing time is entered into the data.
[0017] Preferably, the MiNiFi unit generates a heartbeat regularly and sends it to the NiFi unit, wherein the heartbeat includes a timestamp, an ICU device operating status, memory usage, data collection duration, instantaneous flow, and a unique identifier corresponding to the ICU device and its sensing terminal.
[0018] Preferably, the MiNiFi unit sends a heartbeat every 5 seconds, and the cloud NiFi unit keeps counting down for 10 seconds. If the heartbeat of the MiNiFi unit is received within 10 seconds, it is considered that the connection between the MiNiFi unit and the cloud NiFi unit is normal. If the heartbeat is not received within 10 seconds, the counter is increased by 1; if the value of the counter is greater than or equal to 2, the perception terminal of the ICU device is marked as abnormal.
[0019] The NiFi-based medical device IoT stream data management method of this application can efficiently and flexibly access and manage data to achieve seamless integration and sharing of data, thereby improving the quality of medical services. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 This is the architecture diagram of the real-time data stream medical equipment terminal management system based on NiFi.
[0021] Figure 2 A flowchart for implementing ICU equipment access based on the stream data management method.
[0022] Figure 3 Implement a flow chart for patient information matching based on streaming data management approach.
[0023] Figure 4 Implement a flow chart for data access of imaging equipment based on the streaming data management method; Figure 5 A flow chart of the data traceability and status monitoring implementation method based on MiNiFi-NiFi; Figure 6 This is an architecture diagram based on the MiNiFi-NiFi architecture.
[0024] Figure 7 This is the flow chart of MiNiFi-NiFi heartbeat mechanism. DETAILED DESCRIPTION
[0025] like Figure 1 As shown in the figure, this system builds a data collection system with NiFi as the core, which can realize real-time data collection across hospitals.
[0026] In the process of building a stream data management system, we integrated key technologies such as NiFi, Kafka, Druid, FlinkSQL and MiNiFi to achieve efficient data processing and analysis.
[0027] NiFi: As a core component, it is responsible for comprehensive management of data flows. Through a user-friendly graphical interface, the design, monitoring and management of data flows become intuitive and easy to understand. It can connect to a variety of data sources and support real-time data processing and conversion, and flexibly respond to changes in demand. NiFi's cluster mode ensures high reliability and scalability, enabling the system to handle large-scale data flows.
[0028] Kafka: As a distributed stream processing platform, it acts as a message queue in the system and is responsible for the transmission and buffering of real-time data. Its high throughput and low latency characteristics ensure the rapid flow of data between various components, providing a solid foundation for subsequent data analysis and processing.
[0029] Druid: is a high-performance database focused on real-time data analysis, capable of quickly querying and analyzing large-scale data sets. In our system, Druid is mainly used to ingest data from Kafka in real time, supporting low-latency data query and OLAP (online analytical processing), helping users get the insights they need instantly.
[0030] FlinkSQL: It is a SQL query language for stream processing and batch processing provided by Apache Flink, designed for large-scale data analysis. In this system, FlinkSQL is used to calculate HIS data in real time and generate patient information tables, thereby providing timely and accurate data support for medical staff and promoting the optimization of clinical decision-making.
[0031] MiNiFi: As a lightweight data collection and processing framework, it is mainly deployed on perception terminals close to data sources and is responsible for preliminary data cleaning and processing. It simplifies the data collection process of edge devices and ensures that data can flow efficiently under centralized monitoring and configuration distribution, thereby improving the overall flexibility and response speed of the system.
[0032] like Figure 2 As shown, the implementation of ICU device access based on the stream data management method is explained.
[0033] ICU data is pushed to the sensing terminal by the device through the serial port or network port, and collected and preliminarily processed by MiNiFi on the sensing terminal. MiNiFi aggregates and pushes the data to NiFi in the cloud, and uses the NiFi processor developed by the extension to receive and process the data, including the custom HTTP processor when receiving. Because the HL7 pushed by the device uses custom special characters as delimiters, there is a special character in the header and tail of the HL7 message, so that separate HL7 messages can be separated from continuous data during parsing. HL7 messages are divided into vital signs, waveforms, and alarms. Each type of data needs to be distinguished and processed using different methods. Different manufacturers use different HL7 identifiers, and different manufacturers use different identifiers. The parsing program needs to be able to recognize different identifiers at the same time.
[0034] like Figure 3 As shown, the patient information matching implementation based on the stream data management method is explained.
[0035] In order to solve the problem of ICU data without patient information, we will fill in the patient information in real time when parsing HL7. The specific steps are as follows: first, get the patient information from HIS. Because there is no ready-made table of bed number corresponding to patient information in the HIS system, we must first use FlinkSQL to perform aggregation calculations, aggregate a table containing bed number and patient ID and a table containing patient ID and patient information in real time, and obtain a table of bed number corresponding to patient information, write it into MySQL, and when the HL7 parser parses it, it will query this table in real time to obtain patient information for filling.
[0036] like Figure 4 As shown, the implementation of image device data access based on the stream data management method is explained.
[0037] The access of imaging device data is divided into two parts. The first is the access of real-time data. The real-time data is first pushed to NiFi on the device server. NiFi on the device cleans and converts the data, and then transmits it to NiFi on the cloud through HTTP and pushes it into Kafka. The second is the access of log file data. The device pushes the generated log files to the FTP shared directory on the device server. The log file parser on NiFi on the cloud monitors the changes of files in the FTP shared directory. Once a new log file is found, it obtains the file for parsing and then pushes the parsed data into Kafka.
[0038] FTP shared directory is a network directory provided by the File Transfer Protocol (FTP), which allows users to upload, download and manage files between different computers. It is usually used to conveniently share files in a local area network or the Internet, supports multi-user access and provides corresponding permission settings to ensure data security and controllability.
[0039] The problem of patient information in imaging device data can also be solved through FlinkSQL. The process is similar to the process of obtaining patient information through FlinkSQL when using ICU data.
[0040] like Figure 5 , 6 As shown in ,7, the data traceability and status monitoring implementation method based on MiNiFi-NiFi is described.
[0041] The device pushes data to the collection program, and the collection program packages the data and sends it to MiNiFi. At this time, the data does not carry the unique identifier of the device. Therefore, in order to standardize and flexibly use it, the solution of marking the data with the unique identifier of the device on the perception terminal is adopted. The unique identifier of the device will be set in advance when configuring MiNiFi. The data transmitted by the device will only contain the indicators that need to be collected and the data generation timestamp. After receiving the data, the perception terminal will enter the unique identifier of the device into the data, so that the data can be traced. When NiFi receives the data and processes it, it will also enter the processing time into the data. In this way, if the time when the device generates the data and the time when the data is processed differs greatly, there may be a backlog of data, and it can be specifically known which link has a problem and needs to be processed in time.
[0042] The heartbeat generated by MiNiFi contains timestamp, data perception terminal operation status, memory usage, data collection duration, instantaneous flow, and unique identification of devices and collectors. MiNiFi will send a heartbeat every 5 seconds, and NiFi will count down for 10 seconds. If the heartbeat of MiNiFi is received within 10 seconds, the connection is considered normal. If the heartbeat is not received within 10 seconds, the counter will be increased by 1. If the value of the counter is greater than or equal to 2, that is, more than 20 seconds, the perception terminal will be marked as abnormal. NiFi will summarize and count the data in the heartbeat package and write it to MySQL to provide it to third-party systems for display.
[0043] The NiFi-based medical device IoT streaming data management method of this application, by establishing a unified streaming data management platform, each hospital can collect and manage data under the same framework, thereby eliminating the problem of duplicate construction. Resource sharing and collaboration between hospitals are enhanced, so that data can flow efficiently, reduce the phenomenon of information islands, and greatly improve the efficiency of the use of scientific research funds. By establishing a unified streaming data management platform, each hospital and department can collect and manage data under the same framework, thereby eliminating the problem of duplicate construction. Resource sharing and collaboration between hospitals are enhanced, so that data can flow efficiently, reduce the phenomenon of information islands, and greatly improve the efficiency of the use of scientific research funds. Implement a unified data collection standard to ensure that the data format and structure of different hospitals and departments are consistent. This not only simplifies the process of data integration, reduces the workload of data cleaning and conversion, but also improves the comparability of data, promotes the possibility of cross-hospital cooperation and research, and makes scientific research results more reliable.
[0044] Patient information is matched in real time, combining patient information with data obtained by the device through automated means to ensure that each piece of data has corresponding patient information. This matching mechanism not only improves the accuracy of data analysis, but also enhances the effectiveness of data application, making medical decisions more accurate, improving the overall quality of medical services, and greatly increasing the value of data.
[0045] Use MiNiFi to implement centralized monitoring and configuration distribution of perception terminals, so that the configuration of multiple devices in multiple locations can be modified with one modification. The configuration file can be modified remotely without the need for operation and maintenance personnel to go to the location to make modifications every time.
[0046] Unless otherwise defined, all technical and / or scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which the invention relates. The materials, methods and examples mentioned in this application are illustrative only and not limiting.
[0047] Although the present invention has been described in conjunction with specific embodiments, within the spirit of the invention of this application, those skilled in the art may make appropriate substitutions, modifications and changes, and such substitutions, modifications and changes still fall within the scope of protection of this application.
Claims
1. A medical device IoT stream data management method based on NiFi, which implements stream data management through NiFi unit, Kafka unit, Druid unit, FlinkSQL unit, and MiNiFi unit; wherein, The NiFi unit includes a cloud-side NiFi unit and an imaging device-side NiFi unit. The cloud-side NiFi unit is deployed on the cloud server and is responsible for overall management of data flows, including an ICU data parser and an imaging device data parser. The FlinkSQL unit calculates HIS data in real time and generates patient information for real-time query by the cloud-based NiFi unit; The MiNiFi unit is arranged on a sensing terminal connected to the ICU device, processes the data of the ICU device to obtain processed ICU device data, and the processed ICU device data is sent to the cloud NiFi unit; The NiFi unit on the imaging device side is deployed on the edge server to which the imaging device is connected, and the data of the imaging device is sent to the NiFi unit on the cloud side; The ICU data parser of the cloud NiFi unit processes the processed data from the MiNiFi unit, matches it with the patient information obtained by the real-time query, and obtains the matched ICU equipment data; The imaging device data parser of the cloud-based NiFi unit processes the imaging device data from the NiFi unit on the imaging device side, matches it with the patient information obtained by the real-time query, and obtains the matched imaging device data; The matched ICU device data and the matched imaging device data are pushed to the Kafka unit for the Druid unit to use to ingest data from the Kafka unit in real time; The Druid unit is used to realize real-time display and aggregate calculation of data, as well as to provide external data access services.
2. According to claim 1, the method for managing medical device Internet of Things stream data based on NiFi is characterized in that: The MiNiFi unit on the sensing terminal uses its own extended NiFi processor to aggregate and push the collected and preliminarily processed ICU equipment data to the cloud NiFi unit.
3. The method for managing medical device IoT stream data based on NiFi according to claim 2 is characterized in that: The MiNiFi unit on the sensing terminal communicates with the NiFi unit on the cloud using HL7 messages.
4. The method for managing medical device IoT stream data based on NiFi according to claim 1, characterized in that: When the ICU data parser of the cloud-based NiFi unit parses the ICU equipment data in the form of HL7 messages, it will fill in the parsed data in real time according to the corresponding patient information including the bed number obtained by the aggregation calculation of the FlinkSQL unit.
5. The method for managing medical device IoT stream data based on NiFi according to claim 1, characterized in that: After data cleaning and conversion, the NiFi unit on the imaging device side transmits the imaging device data to the NiFi unit on the cloud side via HTTP.
6. The method for managing medical device IoT stream data based on NiFi according to claim 1, characterized in that: The cloud-based NiFi unit includes a log file parser; the log file parser monitors the log files in the FTP shared directory set by the imaging device, obtains and parses the new log file when a new log file is found, and then pushes the parsed data into the Kafka unit.
7. The method for managing medical device IoT stream data based on NiFi according to claim 1, characterized in that: The ICU device data from the MiNiFi unit includes the metrics that need to be collected, the timestamp of the data generation, and the unique identifier corresponding to the ICU device and its sensing terminal.
8. The method for managing medical device IoT stream data based on NiFi according to claim 1, characterized in that: When the cloud-based NiFi unit processes the received ICU device data, it embeds the processing time into the data.
9. The method for managing medical device IoT stream data based on NiFi according to claim 1, characterized in that: The MiNiFi unit generates a heartbeat periodically and sends it to the NiFi unit. The heartbeat contains a timestamp, ICU device operating status, memory usage, data collection duration, instantaneous traffic, and a unique identifier corresponding to the ICU device and its sensing terminal.
10. The method for managing medical device IoT stream data based on NiFi according to claim 9, characterized in that: The MiNiFi unit sends a heartbeat every 5 seconds, and the cloud NiFi unit keeps counting down for 10 seconds. If the heartbeat of the MiNiFi unit is received within 10 seconds, it is considered that the connection between the MiNiFi unit and the cloud NiFi unit is normal. If the heartbeat is not received within 10 seconds, the counter is increased by 1. If the value of the counter is greater than or equal to 2, the sensing terminal of the ICU device is marked as an abnormal state.