Medical data processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a medical data processing method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a current data offset of corresponding medical data from a message queue; acquiring data index information corresponding to the current data offset; based on the data index information, reading persistently stored medical data; and sending the read persistently stored medical data to a data receiver. By adopting the method, the data can be prevented from being lost, and the reading efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a medical data processing method, device, equipment, storage medium and program product. Background Art

[0002] With the development of smart healthcare, more and more medical data is generated. When requesting and processing medical data, especially when processing high-throughput data, data is easily lost.

[0003] For example, combined with Figure 1 Multiple patient terminals send medical data to the big screen, but 10,000 patients send online messages, but the big screen only shows 9,900 patients online, and 100 patients' online messages are lost, resulting in only 9,900 patients being online on the big screen, or 10,000 patients send offline messages, but many patients are still on the big screen. Figure 2 is a normal ECG waveform of patient 2 in one embodiment, but Figure 3 The ECG waveform data of patient 2 is lost, which results in errors in the final ECG waveform data. Summary of the invention

[0004] Based on this, it is necessary to provide a medical data processing method, device, equipment, storage medium and program product that can ensure that data is not lost and improve reading efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a medical data processing method, the method comprising:

[0006] Get the current data offset of the corresponding medical data from the message queue;

[0007] Obtaining data index information corresponding to the current data offset;

[0008] Based on the data index information, reading the persistently stored medical data;

[0009] The read persistently stored medical data is sent to the data recipient.

[0010] In one embodiment, the method further comprises:

[0011] Receiving medical data to be processed sent by the data sender;

[0012] Persistently storing the medical data to be processed in an asynchronous manner;

[0013] Obtaining the data offset of the persistent storage of the medical data to be processed, and storing the data offset in the corresponding message queue;

[0014] Based on the attributes of the medical data to be processed, data index information of the medical data to be processed is calculated, and the data index information is stored.

[0015] In one embodiment, before the to-be-processed medical data is persistently stored, the method further includes:

[0016] Open a transaction corresponding to the medical data to be processed.

[0017] In one embodiment, after the to-be-processed medical data is persistently stored, the method further includes:

[0018] When the pending medical data is successfully persistently stored, submitting the transaction to the data sender;

[0019] When the persistent storage of the to-be-processed medical data fails, transaction rollback information is sent to the data sender.

[0020] In one embodiment, after storing the data index information, the method further includes:

[0021] Back up the persistently stored medical data, message queues, and data index information to a backup server.

[0022] In one embodiment, obtaining the current data offset of the medical data corresponding to the data receiver from the message queue includes:

[0023] Obtain the topic corresponding to the data recipient, and determine the message queue corresponding to the topic;

[0024] The current data offset of the medical data corresponding to the data receiver is obtained from the determined message queue.

[0025] In one embodiment, after sending the read persistently stored medical data to the data recipient, the method includes:

[0026] Updates the current data offset in the message queue.

[0027] In one embodiment, obtaining the current data offset of the medical data corresponding to the data receiver from the message queue includes:

[0028] Obtain the current data offset of the medical data corresponding to the data receiver from the message queue stored in the memory;

[0029] The acquiring of data index information corresponding to the current data offset includes:

[0030] The data index information corresponding to the current data offset is obtained from the index information file stored in the memory.

[0031] In one embodiment, reading the persistently stored medical data based on the data index information includes:

[0032] Based on the data index information, medical data is read from the memory, where the medical data stored in the memory is obtained by processing the medical data persistently stored in the disk based on the memory mapping method;

[0033] When the medical data is not read from the memory, the persistently stored medical data is read from the disk based on the data index information.

[0034] In one embodiment, reading the persistently stored medical data from the disk includes:

[0035] Generate system read and write scheduling tasks based on data index information of the medical data;

[0036] Scheduling the scheduling task according to the system read-write scheduling algorithm;

[0037] When the scheduling task is scheduled, the persistently stored medical data is read from the disk based on the scheduling task.

[0038] In a second aspect, the present application further provides a medical data processing device, the device comprising:

[0039] A current data offset acquisition module is used to obtain the current data offset of the medical data corresponding to the data receiver from the message queue;

[0040] An index information acquisition module, used to acquire data index information corresponding to the current data offset;

[0041] A data reading module, used for reading the persistently stored medical data based on the data index information;

[0042] The consumption module is used to send the read persistently stored medical data to the data recipient.

[0043] In one embodiment, the device further comprises:

[0044] A data receiving module, used for receiving the medical data to be processed sent by the data sender;

[0045] A persistent storage module, used for persistently storing the medical data to be processed in an asynchronous manner;

[0046] A message queue storage module, used to obtain the data offset of the persistent storage of the medical data to be processed, and store the data offset in the corresponding message queue;

[0047] The index information storage module is used to calculate the data index information of the medical data to be processed based on the attributes of the medical data to be processed, and store the data index information.

[0048] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above-mentioned embodiments when executing the computer program.

[0049] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method in any one of the above-mentioned embodiments.

[0050] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any one of the above-mentioned embodiments when executed by a processor.

[0051] The above-mentioned medical data processing method, device, equipment, storage medium and program product store medical data persistently, thus ensuring that the data is not lost. In medical data processing, the current data offset is obtained from the message queue, and data index information is obtained based on the current data offset. In this way, medical data is read based on the data index, which improves reading efficiency and reduces the pressure caused by overall network traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of medical data displayed on a large screen;

[0053] Figure 2 is a normal electrocardiogram waveform of patient 2 in one embodiment;

[0054] Figure 3 is an electrocardiogram waveform diagram when the electrocardiogram waveform data of patient 2 is lost in one embodiment;

[0055] Figure 4 is an application environment diagram of a medical data processing method in one embodiment;

[0056] Figure 5 A structural block diagram of a proxy server in one embodiment;

[0057] Figure 6 is a flowchart of a medical data processing method in one embodiment;

[0058] Figure 7is an ECG waveform diagram of a patient 2 when the ECG waveform data is out of order in one embodiment;

[0059] Figure 8 This is an ECG waveform diagram of the patient 2 when the ECG waveform data is stuck in one embodiment;

[0060] Fig. 9 A schematic diagram of a medical data storage step in an embodiment;

[0061] Fig.10 is an architectural diagram of a medical data processing method in one embodiment;

[0062] Fig.11 is a flow chart of a medical data processing method in another embodiment;

[0063] Fig.12 is a structural block diagram of a medical data processing device in one embodiment;

[0064] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The medical data processing method provided in the embodiment of the present application can be applied to Figure 4 In the application environment shown, the data sender 402 and the data receiver 406 communicate with the intermediate system 404 through the network. The data storage system can store the data that the intermediate system 404 needs to process, and can also be placed on the cloud or other network servers.

[0067] The data sender 402 is the producer of medical data, the data receiver 406 is the consumer of medical data, and the intermediate system 404 is the middleware between the data sender 402 and the data receiver 406, which is used to receive the medical data sent by the data sender 402 and store it in the topic and message queue inside the intermediate system 404. The intermediate system 404 is used to receive, persist and manage medical data so that subsequent data receivers can obtain and process the medical data.

[0068] Among them, the data sender 402 is used to create medical data, which can be text, JSON, binary data or other formats. The data sender sends the created medical data to the message queue. In some optional embodiments, the process of sending medical data is usually asynchronous, that is, after the data sender 402 sends the medical data to the message queue, it can immediately continue to perform other operations without waiting for the processing results of the medical data. After the data sender 402 accesses the intermediate system 404 and obtains the list of proxy servers in the intermediate system 404, the medical data can be delivered through the load balancing module of the intermediate system 404. This process is asynchronous and supports fast failure, such as network connection interruption, message queue unavailability, etc. Error handling strategies can also be set, such as retry and delayed retry. At the same time, the performance indicators and sending status of the data sender 402 can be monitored and recorded for troubleshooting and performance tuning.

[0069] The data receiver 406 is responsible for the consumption of medical data. It can consume medical data through various modes, including but not limited to the proxy server actively pushing medical data to the data receiver 406 and the data receiver 406 actively pulling medical data. In addition, the data receiver also supports cluster and broadcast consumption. It provides a real-time message subscription mechanism, enabling the data receiver 406 to receive and process medical data in real time. The cluster mode means that a message will be distributed to one of the data receivers 406 in the corresponding consumer group according to the load balancing value, which is an average distribution form. The broadcast mode means that a message will be distributed to all data receivers 406 in the corresponding consumer group, which is a full consumption form.

[0070] The proxy server is in the intermediate system 404, which is used to receive medical data from the data sender 402 and store it in internal topics and queues. It is responsible for receiving, persisting and managing these messages so that the subsequent data receiver 406 can obtain and process them.

[0071] Secondly, the proxy server provides an API interface to connect with the data sender 402 and the data receiver 406. Through these API interfaces, the data sender 402 can send medical data to the proxy server, and the data receiver 406 can subscribe to and obtain medical data from the proxy server. In addition, the proxy server also has the ability to persist medical data, that is, to store medical data in a persistent storage medium to ensure the reliability of medical data after system failure or restart. Therefore, the proxy server assumes multiple roles such as storage, persistence, and interface docking, and is an important component and service provider in the message queue system.

[0072] Specifically, combined Figure 5As shown, the proxy server may include a remote communication module, a client management module, a storage module, a high availability module, and an index module. The remote communication module is responsible for processing requests from the data sender 402 and the data receiver 406, assumes the role of network communication, is responsible for establishing connections with the data sender 402 and the data receiver 406, receives and parses requests, and sends requests to other modules of the proxy server. The client management module is responsible for managing the client, including the data sender 402 and the data receiver 406, and maintaining the topic subscription information. The client management module tracks and manages the client connected to the proxy server, records the registration information of the client, and maintains the topic subscription relationship of the data receiver 406 to ensure that the medical data can be correctly routed and delivered to the data receiver 406 who has subscribed to the relevant topic. The client management module acts as a bridge between the client and the proxy server, coordinates the registration and subscription behavior of the client, and provides a real-time subscription mechanism so that the client can receive the required medical data in a timely manner. The storage module is responsible for persistently storing the received medical data on disk and providing an efficient query interface for rapid retrieval and access to the stored medical data. The storage module simplifies the operation of medical data storage and query, allowing developers to easily use API interfaces to handle the persistence and retrieval of medical data, improving the reliability and performance of the system. The high availability module is responsible for providing data synchronization between the main proxy server and the backup server. The high availability module monitors the status of the main proxy server and synchronizes the medical data to the backup server in real time to maintain data consistency between the two. When the main proxy server fails or is unavailable, the high availability module is responsible for switching to the backup server, enabling the system to achieve high availability and fault tolerance, and ensure the continuity and reliability of the message queue. The index module provides efficient medical data query functions for the message queue system. By establishing an index structure, the key information of the medical data is mapped to its location in the storage to achieve fast medical data positioning and fast query.

[0073] A topic is a topic in a proxy server. A topic can be stored in multiple proxy servers, and a proxy server can also store multiple topics. A topic represents a class of related messages. The data sender 402 publishes medical data to a specific topic, and the data receiver 406 can subscribe to the topic of interest to receive the corresponding medical data. In this application, topics can be divided according to the type of medical equipment. For example, medical equipment includes but is not limited to ECG equipment, blood oxygen equipment, blood pressure equipment, and temperature equipment. The corresponding topics include ECG data topics, blood oxygen data topics, blood pressure data topics, and temperature data topics.

[0074] In some optional embodiments, the application environment may also include a registration center, which may support dynamic registration of a proxy server and store a mapping relationship between a subject and a proxy server, so that the data receiver 406 and the data sender 402 may obtain the mapping relationship from the registration center, find the proxy server where the corresponding subject is located based on the mapping relationship, and obtain the routing information of the proxy server. The proxy server will register with the registration center, and the registration center provides a heartbeat check mechanism to check whether the proxy server is down. If the proxy server is down, the proxy server will be deleted from the registration table of the registration center, so that the data sender 402 and the data receiver 406 can obtain the registration table from the registration center, and obtain the routing information of the proxy server based on the registration table, so as to facilitate the sending and receiving of medical data. The registration center is deployed in a cluster manner. The proxy server registers its own routing information with each server in the registration center, so a complete routing information is stored on the service of each server in the registration center. If a server in a registration center goes offline, the proxy server can synchronize its routing information with other servers in the registration center, and the data sender 402 and the data receiver 406 will also dynamically perceive the changes in the registration table of the proxy server.

[0075] Among them, the intermediate system 404 can obtain the current data offset of the corresponding medical data from the message queue; obtain the data index information corresponding to the current data offset; read the persistently stored medical data based on the data index information; and send the read persistently stored medical data to the data receiver 406. In this way, the persistent storage of medical data ensures that the data is not lost, and in the medical data processing, the current data offset is obtained from the message queue, and the data index information is obtained based on the current data offset. In this way, the medical data is read based on the data index, which improves the access efficiency and reduces the pressure caused by the overall network traffic.

[0076] The data sender 402 and the data receiver 406 can be implemented by terminals, which can be, but are not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The intermediate system 404 can be implemented by an independent server or a server cluster consisting of multiple servers.

[0077] In one embodiment, Figure 6 As shown, a medical data processing method is provided, which is applied to Figure 4 Taking the intermediate system 404 in the example as an example, the following steps are included:

[0078] S602: Obtain the current data offset of the corresponding medical data from the message queue.

[0079] Specifically, the message queue is in the intermediate system, and the message queue is used to store the consumption progress of the medical data by the data receiver. The message queue is used to record the consumption progress of the medical data of each data receiver on a specific topic or queue. Optionally, the message queue stores the offset of the last medical data consumed by the data receiver, so that the data receiver can continue to consume accurately after restarting or recovering from a fault. The data receiver can find the medical data to be processed according to the message queue. In some optional embodiments, the message queue serves as an index of the medical data of the data receiver, and stores the starting physical offset offset of the medical data under a specific topic in the persistent storage, the message size size and the HashCode value of the message Tag. The message queue file can be regarded as an index file of a persistent storage file based on topic, so the organization of the message queue folder is as follows: a three-layer organizational structure of topic / queue / file, and the specific storage path is: $HOME / store / consumequeue / {topic} / {queueId} / {fileName}. When the data sender sends medical data to the proxy server, the medical data will be written into the persistent storage file first. The proxy server will generate a unique message tag for the medical data, associate the message tag with the HashCode value of the medical data, and store it in the message queue file. The data recipient can quickly locate the corresponding medical data storage location in the persistent storage file through the recorded message tag and HashCode value.

[0080] The offset is the location of the medical data in the persistent storage. Each medical data has a unique offset that identifies the storage location of the message in the persistent storage. The offset is usually an integer value that can be used to quickly locate and find medical data. With the offset, medical data in the persistent storage can be read and retrieved sequentially.

[0081] The current data offset is the offset of the last medical data consumed by the data receiver stored in the message queue, or the offset of the next medical data to be consumed by the data receiver according to the data sequence. The offset of the medical data in the message queue is stored in a certain order, which can avoid the following problems: Figure 7 The disorder of data in Figure 7 The data of the third package is drawn before the data of the second package, resulting in an incorrect waveform presented to the doctor. In this embodiment, the offsets of the medical data in the message queue are stored in sequence, which avoids data disorder and ensures that data is consumed in sequence.

[0082] In one embodiment, obtaining the current data offset of the medical data corresponding to the data recipient from the message queue includes: obtaining the topic corresponding to the data recipient and determining the message queue corresponding to the topic; obtaining the current data offset of the medical data corresponding to the data recipient from the determined message queue.

[0083] Specifically, the proxy server stores the relationship between the data recipient and the topic, which is generated when the data recipient subscribes to the topic. There is also a mapping relationship between the topic and the message queue, so the message queue corresponding to the data recipient can be obtained, thereby obtaining the current data offset of the medical data corresponding to the data recipient from the message queue.

[0084] S604: Obtain data index information corresponding to the current data offset.

[0085] Specifically, the index information is the index information of medical data, which supports the rapid query of medical data. The data index information can be stored in the index information file. The index information is an auxiliary data structure of persistent storage, which is used to accelerate the search and retrieval of medical data. The index information records the key attributes of the message (such as the subject, queue and offset of the medical data, etc.) and maps it to the location where the actual medical data is stored. Through the index information, the medical data in the persistent storage can be quickly located according to the key attributes of the medical data, thereby improving the retrieval efficiency of the medical data.

[0086] The storage location of the index file is: $HOME\store\index${fileName}, and the file name fileName is named after the timestamp when it was created. In an optional embodiment, the fixed single IndexFile file size is about 400M, and an IndexFile can save 20 million indexes. The underlying storage of IndexFile is designed to implement the HashMap structure in the file system, so the underlying implementation of the index information file is a hash index. The introduction of the index information file can ensure that the medical data can be consumed in time, thereby solving the following problems: Figure 8 The problem of data processing lag shown, for example, the problem of waveform drawing lag has been solved.

[0087] S606: Read the persistently stored medical data based on the data index information.

[0088] Specifically, the index information may include attributes of medical data, such as the subject, queue and offset of the medical data, so that the proxy server of the intermediate system can determine the corresponding data index information based on the current data offset, and quickly obtain the key attributes of the medical data based on the index information, thereby locating the medical data in the persistent storage and improving the retrieval efficiency.

[0089] The persistent storage is to persistently store the medical data sent by the data sender, and maintain the offset and data index information of the medical data in the persistently stored log file to support fast data retrieval, wherein the data index information can be established according to the attributes of the medical data, such as key information, so that the query and reading operations of the medical data are more efficient, thereby ensuring that the medical data in the queue is consumed in time and that the ecg waveform is drawn continuously. In one of the optional embodiments, the single file size of the persistently stored log file defaults to 1G, the file name length is 20 bits, zeros are padded on the left, and the rest is the starting offset, for example, 0000000000000000000 represents the first file, the starting offset is 0, and the file size is 1G=1073741824; when the first file is full, the second file is 00000000001073741824, and the starting offset is 1073741824, and so on. Medical data is written into log files in sequence. When a file is full, it is written into the next file, thus ensuring the sequential consumption of medical data.

[0090] S608: Send the read persistently stored medical data to a data recipient.

[0091] After obtaining the medical data, the proxy server of the intermediate system sends the obtained medical data to the data receiver. In this embodiment, the data receiver's consumption of data includes active push by the proxy server and active pull by the data receiver. However, whether it is active push or active pull, the retrieval of medical data first obtains the current data offset from the message queue, and obtains the data index information based on the current data offset, so that the medical data can be read based on the data index, thereby improving the efficiency of retrieval.

[0092] In one of the embodiments, after the read persistently stored medical data is sent to the data receiver, the method includes: updating the current data offset in the message queue.

[0093] Specifically, the data offset of the consumed medical data is removed from the message queue to avoid re-consumption of the medical data.

[0094] The above-mentioned medical data processing method stores medical data persistently, which ensures that the data is not lost. In the medical data processing, the current data offset is obtained from the message queue, and the data index information is obtained based on the current data offset. In this way, medical data is read based on the data index, which improves access efficiency and reduces the pressure caused by the overall network traffic.

[0095] In one embodiment, see Fig. 9 As shown, Fig. 9Schematic diagram of a medical data storage step in an embodiment, in which the medical data storage step includes:

[0096] S902: Receive the medical data to be processed sent by the data sender.

[0097] The data sender creates medical data and sends the medical data to be processed to the intermediate system. Specifically, the data sender can be an ECG device, a blood oxygen device, a blood pressure device, a temperature device, etc., and the corresponding medical data to be processed are ECG data, blood oxygen data, blood pressure data, and temperature data, etc.

[0098] Before the data sender sends the medical data to be processed, the data sender first sends a message to the intermediate system. The message can be the type of the data sender, that is, first establish a connection with the registration center of the intermediate system and send a message to the registration center. The registration center is responsible for maintaining the metadata information of the entire intermediate system cluster, including the routing information of the proxy server and the subscription relationship of the subject, etc. After the registration center receives the message sent by the data sender, it selects the corresponding proxy server according to the subject of the message and the load balancing strategy, and sends the routing information of the selected proxy server to the data sender. After the data sender receives the routing information of the proxy server, it establishes a connection with the corresponding proxy server and sends the medical data to be processed to the remote communication module of the proxy server. The remote communication module receives the medical data to be processed and can also perform simple processing on the medical data to be processed, such as decoding, etc., which is not specifically limited here.

[0099] S904: The medical data to be processed is persistently stored in an asynchronous manner.

[0100] The data sender sends the pending medical data asynchronously, that is, after the data sender sends the pending medical data, other operations can be performed immediately. The proxy server stores the pending medical data persistently, for example, persistently in a log file. In this embodiment, the pending medical data adopts a hybrid storage structure. All message queues under a single instance of the proxy server share a log file to store the pending medical data. The data sender sends the pending medical data to the proxy server, and then the proxy server uses an asynchronous method to persist the pending medical data on the disk and save it in the log file. As long as the pending medical data is persisted on the disk in the log file, the pending medical data sent by the data sender will not be lost. Because of this, the data recipient will definitely have the opportunity to consume this pending medical data. When the pending medical data cannot be pulled, you can wait for the next time to pull the pending medical data. At the same time, the intermediate system also supports long polling mode. If a pending medical data pull request does not pull the pending medical data, the proxy server allows waiting for a certain period of time, such as 30s. As long as new pending medical data arrives during this period, it will be directly returned to the data recipient.

[0101] S906: Obtain the data offset of the persistent storage of the medical data to be processed, and store the data offset in the corresponding message queue.

[0102] The data offset can be seen above, that is, the data offset of the medical data to be processed in the persistent storage, and the data offset is stored in the message queue of the corresponding topic.

[0103] S908: Calculate data index information of the medical data to be processed based on the attributes of the medical data to be processed, and store the data index information.

[0104] Specifically, the attributes of the medical data to be processed may include a subject, a queue, an offset, etc., and data index information of the medical data to be processed is obtained based on these attributes, and the data index information is stored.

[0105] In one of the embodiments, after storing the data index information, it also includes: backing up the persistently stored medical data, message queue and data index information to a backup server.

[0106] Among them, in order to achieve high-availability services, the proxy server corresponds to a backup server, in which the proxy server's persistently stored medical data, message queues, and data index information are backed up to the backup server. In this way, when the proxy server fails or is unavailable, it switches to the backup server, enabling the system to achieve high availability and fault tolerance, and ensure the continuity and reliability of the message queue.

[0107] In the above embodiment, the medical data to be processed is stored on the disk through persistent storage, the message queue records the data offset of the data recipient, and the index information is used to quickly locate the physical location of the medical data to be processed in the persistent storage. In this way, persistent storage ensures that data will not be lost, the message queue ensures the orderliness of data consumption, and the index information improves the efficiency of data consumption.

[0108] In one embodiment, before persistently storing the pending medical data, the method further includes: starting a transaction corresponding to the pending medical data. In one embodiment, after persistently storing the pending medical data, the method further includes: when the persistent storage of the pending medical data is successful, submitting the transaction to the data sender; when the persistent storage of the pending medical data fails, sending transaction rollback information to the data sender.

[0109] In this embodiment, the concept of transaction is introduced, and the medical data to be processed is realized through transactions, which ensures that the medical data to be processed will not be consumed multiple times. That is, when the medical data to be processed is received, the corresponding transaction is started. When the medical data to be processed is successfully persisted and stored, the transaction is submitted to the data sender. If the storage fails, the transaction is rolled back, so that the data sender resends the transaction.

[0110] Specifically, the data sender sends a transaction message, and the proxy server responds to the transaction message and writes the result. The local transaction is executed according to the sent result. If the write fails, the local logic is not executed, and the transaction is committed or rolled back according to the local transaction status. For transaction messages that are not committed or rolled back, a checkback is initiated from the proxy server. The data sender receives the checkback message, checks the status of the local transaction of the data sender corresponding to the checkback message, and resubmits the transaction or rolls back the transaction according to the local transaction status.

[0111] Taking ECG data as an example, the transaction here refers to the delivery of ECG data. When the data sender sends an ECG message to the proxy server, a transaction is started. At this time, it will not be pushed to the data receiver immediately, but will be written to the persistent storage first. After the write is successful, the transaction is committed, indicating that the proxy server has successfully received, recorded, and saved the ECG data. At this time, the data sender will not deliver the ECG data again, and the corresponding data receiver can consume the ECG data; when the write fails, the transaction is rolled back, indicating that the proxy server has not successfully received the ECG data. At this time, the data sender will deliver the ECG data again, and the ECG data will not be read by the data receiver. This ensures that the ECG data will only be consumed by the data receiver after it is successfully saved to the proxy server.

[0112] The transaction concept is introduced in the above embodiment to ensure that the medical data to be processed will not be consumed multiple times.

[0113] In one of the embodiments, obtaining the current data offset of the medical data corresponding to the data recipient from the message queue includes: obtaining the current data offset of the medical data corresponding to the data recipient from the message queue stored in the memory; obtaining data index information corresponding to the current data offset includes: obtaining the data index information corresponding to the current data offset from the index information file stored in the memory.

[0114] Specifically, the page cache (PageCache) is the OS's cache of files, which is used to speed up the reading and writing of files. Generally speaking, the speed at which a program sequentially reads and writes files is almost close to the speed at which the memory reads and writes. The main reason is that the OS uses the PageCache mechanism to optimize the performance of read and write access operations, and uses part of the memory as PageCache. For data writing, the OS will first write it to the cache, and then the pdflush kernel thread will flush the data in the cache to the physical disk in an asynchronous manner. For data reading, if the PageCache is not hit when reading a file once, the OS will sequentially pre-read the data files of other adjacent blocks while accessing the read file from the physical disk.

[0115] In this embodiment, since the data stored in the message queue and index information file is relatively small, and the data in the message queue is stored sequentially, in order to improve processing efficiency, the message queue and index information file are stored in the memory. In this way, the reading performance of the message queue and index information file is almost close to that of the memory, and the performance will not be affected even if there is message accumulation.

[0116] In one of the embodiments, reading persistently stored medical data based on data index information includes: reading medical data from a memory based on the data index information, the medical data stored in the memory being obtained by processing the medical data persistently stored on a disk based on a memory mapping method; when the medical data is not read from the memory, reading the persistently stored medical data from the disk based on the data index information.

[0117] Specifically, in order to improve the reading efficiency of data in persistent storage, memory mapping is introduced, such as MappedByteBuffer. MappedByteBuffer is a memory mapping method for file storage, which can map files directly to memory, thereby realizing direct reading and writing operations on files. Priority processing logic is used to directly map physical files in the disk to user-state memory addresses. This Memory Mapping method avoids the copy overhead of data between the kernel buffer and the user application buffer in traditional IO, and converts the operation of files into direct operation on memory addresses, thereby greatly improving the reading and writing efficiency of files. In order to facilitate mapping the entire file to memory at one time, file storage is suitable for fixed-length structure storage. Specifically, since the priority processing logic uses a buffer to read and write data, it is necessary to process the boundaries and integrity of the data. At the same time, when using the priority processing logic, it is necessary to pre-allocate the buffer and memory space, so it may occupy more memory resources. Therefore, fixed-length file storage can more conveniently handle data boundary issues and more effectively manage memory resources.

[0118] In one embodiment, mapping the persistent storage file to the user state memory address can be implemented by using the Java MappedByteBuffer class. The specific steps are as follows:

[0119] First, open the persistent storage file: use Java's FileChannel class to open the persistent storage file and obtain the file size.

[0120] Second, create a MappedByteBuffer: Use the map method of FileChannel to map the persistent storage file to the user-state memory address and create a MappedByteBuffer object.

[0121] Read and write operations: Use MappedByteBuffer objects for read and write operations. Since the data has been mapped to the memory, the read and write operations are equivalent to being performed directly in the memory, which can achieve high read and write performance.

[0122] Asynchronous disk flushing: To ensure persistent storage of data, you can use the asynchronous disk flushing mechanism. By calling the force method of MappedByteBuffer, the data in the buffer is flushed to the persistent storage file on the disk.

[0123] Close MappedByteBuffer: After completing the read and write operations, you need to close the MappedByteBuffer object and release the corresponding memory resources.

[0124] In one of the embodiments, reading persistently stored medical data from a disk includes: generating a system read / write scheduling task based on data index information of the medical data; scheduling the scheduling task according to a system read / write scheduling algorithm; and when the scheduling task is scheduled, reading persistently stored medical data from the disk based on the scheduling task.

[0125] The system IO scheduling algorithm is used to read the medical data in persistent storage. For the log data files in persistent storage, more random access reads will be generated when reading the content, which seriously affects the performance. Here, the system IO scheduling algorithm is selected, such as Deadline. Deadline is a system IO scheduling algorithm that optimizes the order of accessing storage devices according to the deadline of I / O requests to improve response performance and throughput. At this time, SSD is used for block storage to improve the performance of random reads.

[0126] Among them, the introduction of system scheduling algorithms can optimize disk IO access: the deadline scheduling algorithm can reasonably arrange access time in disk IO operations to minimize IO latency. This is very important for reading data from persistently stored files, because persistent storage of messages is usually performed on disks, and optimizing IO access can improve data reading efficiency.

[0127] Introducing a system scheduling algorithm can improve the responsiveness of data reading: the deadline scheduling algorithm can arrange the execution order of IO operations according to the deadline of the IO request to minimize the response time of the IO request. This is especially important for reading data from persistent storage files, because the reading operation of messages usually requires a timely response to meet the needs of real-time data access.

[0128] Introducing a system scheduling algorithm can increase data reading throughput: The Deadline scheduling algorithm can arrange the execution order of IO operations according to the priority of IO requests to maximize data reading throughput. For data reading of persistent storage files, the efficiency and throughput of data reading can be effectively improved by reasonably arranging the execution order of read operations.

[0129] Among them, reading the persistently stored medical data through the system IO scheduling algorithm may include the following steps:

[0130] Submit a read request: The data receiver sends a read request, specifying the offset and size of the message to be read.

[0131] Select the appropriate storage file: The proxy server determines the persistent storage file for the message based on the offset specified in the read request, and then locates the corresponding storage location based on the file's index information.

[0132] Constructing a read task: The proxy server converts the read request into a task of the system IO scheduling algorithm Deadline. The task includes information such as the file to be read, the read position, and the read size.

[0133] Task queuing: The proxy server adds the constructed read task to the task queue of the system IO scheduler and waits for scheduling.

[0134] System IO Scheduling: The system IO scheduler selects the appropriate time to execute tasks according to the deadline mechanism. It dynamically schedules tasks based on factors such as task priority, deadline, and device load to maximize the efficiency of IO operations.

[0135] Reading data: When the task is scheduled for execution, the proxy server will read the medical data through the asynchronous IO interface of the underlying operating system. After the medical data is read, the proxy server returns the read medical data to the data recipient.

[0136] In the above embodiment, the proxy server uses page cache and memory mapping technology to load part of the file data into the memory to improve the read and write performance. Through memory mapping, the file can be read and written directly in the memory, avoiding frequent disk IO operations, thereby improving the message processing efficiency.

[0137] For ease of understanding, combined Fig.10 As shown, Fig.10 The diagram is an architecture diagram of a medical data processing method in an embodiment, which includes an electrophysiological data acquisition client, a system server, and an electrophysiological data consumption client. The electrophysiological data acquisition client may include an electrocardiogram device, a blood oxygen device, a blood pressure device, and a temperature acquisition device, etc. The electrophysiological data consumption client may include an electrophysiological data display client and electrophysiological data storage and function calculation (calculating whether the patient has abnormal heart rhythm data based on the ECG data of the electrocardiogram device, and abnormal heart rhythm includes suspected atrial fibrillation, suspected abnormal rhythm, and suspected interference).

[0138] The electrophysiological data acquisition client in this embodiment is equivalent to the data sender mentioned above, and the electrophysiological data consumption client is equivalent to the data receiver mentioned above. The data receivers here can be divided into four categories: one is a real-time display data receiver, one is a data receiver that stores messages in a wide table, one is a data receiver that processes data into a result through function calculation, for example, the calculation result of this data is suspected atrial fibrillation, and one is a data receiver that integrates the data into a report (the report may include the average heart rate, maximum and minimum values, etc. of this detection).

[0139] Specifically, combined Fig.11 As shown, Fig.11 FIG. 1 is a flow chart of a medical data processing method in another embodiment. The medical data processing method includes:

[0140] First, the electrophysiological data acquisition client sends a message to the registration center of the system server: the electrophysiological data acquisition client establishes a connection with the registration center of the system server and sends the message to the registration center. The registration center is responsible for maintaining the metadata information of the entire proxy server cluster, including the routing information of the proxy server and the subscription relationship of the subject.

[0141] The registration center returns the routing information of the proxy server: After receiving the message from the electrophysiological data acquisition client, the registration center returns the routing information of the selected proxy server according to the subject information of the message and the load balancing strategy.

[0142] The remote communication module of the electrophysiological data acquisition client sending the medical data to be processed to the proxy server: According to the routing information returned by the registration center, the electrophysiological data acquisition client establishes a connection with the selected proxy server and sends the message to the remote communication of the proxy server. The remote communication is responsible for receiving and processing the medical data to be processed sent by the electrophysiological data acquisition client.

[0143] The remote communication module of the proxy server hands over the medical data to be processed to the client management module: After receiving the message, the remote communication module of the proxy server hands over the message to the client management module of the proxy server. The client management module is responsible for managing the client (electrophysiological data acquisition client / electrophysiological data consumption client) and maintaining the subscription relationship between the electrophysiological data consumption client and the topic.

[0144] The client management module passes the message to the storage module: The client management module of the proxy server passes the pending medical data to the storage module of the proxy server. The storage module is responsible for processing the persistent storage of the pending medical data to the physical hard disk and the query function.

[0145] The storage module stores the message in a persistent storage file: The storage module of the proxy server writes the pending medical data into a persistent storage file to ensure the persistent storage of the pending medical data. The persistent storage file is the message storage core in the proxy server.

[0146] The storage module stores the message in the message queue file: At the same time, the storage module also stores the metadata of the pending medical data in the message queue file to support the query of the fast consumption of the pending medical data.

[0147] The storage module stores the message in the index information file: The storage module also stores the medical data to be processed in the index information file according to the specific attributes of the medical data to be processed (such as the Message key) to provide fast index query of the medical data to be processed.

[0148] High availability module performs master-slave replication: If the proxy server uses master-slave mode or cluster mode, the high availability module of the proxy server will perform master-slave replication of the medical data to be processed to ensure redundant backup and high availability of the medical data to be processed.

[0149] Electrophysiological data consumption client consumes pending medical data: Electrophysiological data consumption client can consume pending medical data by subscribing to corresponding topics. Electrophysiological data consumption client establishes a connection with the proxy server and pulls pending medical data for consumption based on the message queue and index information. The consumption method can be active pulling or the proxy server actively pushing to the electrophysiological data consumption client.

[0150] Among them, when the medical data to be processed is consumed by the electrophysiological data consumption client, the electrophysiological data consumption client will read the offset of the medical data to be processed from the message queue and consume it. After the consumption is completed, the offset will be deleted from the message queue. When consuming, based on the offset in the message queue, an index is found in the index information file, and the specific medical data to be processed is found through this index for persistent storage, and then sent to the electrophysiological data consumption client.

[0151] The proxy server uses page cache and memory mapping technology to load part of the file data into the memory to improve the read and write performance. Through memory mapping, the file can be read and written directly in the memory, avoiding frequent disk IO operations, thereby improving the processing efficiency of the medical data to be processed.

[0152] The transaction mechanism of the proxy server is used to ensure the reliability of the medical data to be processed. When the electrophysiological data acquisition client sends medical data, the medical data is first written to the persistent storage, but it is not immediately visible to the electrophysiological data consumption client. The medical data to be processed will not be visible to the electrophysiological data consumption client until the transaction is committed. If the transaction is rolled back, the medical data to be processed will not be consumed, thus ensuring that the medical data to be processed will not be consumed multiple times.

[0153] In the above embodiment, file storage, page cache, memory mapping and transaction mechanism are used to ensure that messages will not be blocked under high concurrency: file storage and memory mapping technology improve the read and write performance of the medical data to be processed, reduce IO overhead, and enable the system to process a large number of message requests without being blocked. The medical data to be processed can be consumed sequentially, wherein the message queue records the consumption offset of the electrophysiological data consuming client, and can be consumed in the order of the medical data to be processed, ensuring that the medical data to be processed is correctly consumed in the order in which it is sent. Avoid multiple consumption of the same medical data: The transaction mechanism ensures that the medical data to be processed will be visible to the electrophysiological data consuming client only after the transaction is committed. If the transaction is rolled back, the medical data to be processed will not be consumed, avoiding the problem of the same medical data to be processed being consumed multiple times.

[0154] In summary, the collaboration of file storage, page cache, memory mapping, and transaction mechanism ensures the medical data processing efficiency, sequential consumption of medical data, and reliability of medical data in high-concurrency scenarios of the proxy server.

[0155] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0156] Based on the same inventive concept, the embodiment of the present application also provides a medical data processing device for implementing the medical data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more medical data processing device embodiments provided below can refer to the limitations of the medical data processing method above, and will not be repeated here.

[0157] In one embodiment, Fig.12 As shown, a medical data processing device is provided, including: a current data offset acquisition module 1201, an index information acquisition module 1202, a data reading module 1203 and a consumption module 1204, wherein:

[0158] The current data offset acquisition module 1201 is used to acquire the current data offset of the medical data corresponding to the data receiver from the message queue;

[0159] An index information acquisition module 1202 is used to acquire data index information corresponding to the current data offset;

[0160] A data reading module 1203 is used to read the persistently stored medical data based on the data index information;

[0161] The consumption module 1204 is used to send the read persistently stored medical data to a data recipient.

[0162] In one embodiment, the above device further comprises:

[0163] A data receiving module, used for receiving the medical data to be processed sent by the data sender;

[0164] A persistent storage module, used for persistently storing the medical data to be processed in an asynchronous manner;

[0165] A message queue storage module, used to obtain the data offset of the persistent storage of the medical data to be processed, and store the data offset in the corresponding message queue;

[0166] The index information storage module is used to calculate the data index information of the medical data to be processed based on the attributes of the medical data to be processed, and store the data index information.

[0167] In one embodiment, the above device further comprises:

[0168] The transaction opening module is used to open the transaction corresponding to the medical data to be processed.

[0169] In one embodiment, the above device further comprises:

[0170] The transaction processing module is used to submit the transaction to the data sender when the persistent storage of the pending medical data is successful; and to send transaction rollback information to the data sender when the persistent storage of the pending medical data fails.

[0171] In one embodiment, the above device further comprises:

[0172] The backup module is used to back up the persistently stored medical data, message queues and data index information to a backup server.

[0173] In one embodiment, the current data offset acquisition module 1201 is also used to acquire the subject corresponding to the data recipient, and determine the message queue corresponding to the subject; and acquire the current data offset of the medical data corresponding to the data recipient from the determined message queue.

[0174] In one embodiment, the above device further comprises:

[0175] An update module is used to update the current data offset in the message queue.

[0176] In one of the embodiments, the current data offset acquisition module 1201 is further used to acquire the current data offset of the medical data corresponding to the data receiver from the message queue stored in the memory;

[0177] The index information acquisition module 1202 is further configured to acquire data index information corresponding to the current data offset from an index information file stored in the memory.

[0178] In one embodiment, the data reading module 1203 is also used to read medical data from the memory based on the data index information, and the medical data stored in the memory is obtained by processing the medical data persistently stored in the disk based on the memory mapping method; when the medical data is not read from the memory, the persistently stored medical data is read from the disk based on the data index information.

[0179] In one of the embodiments, the data reading module 1203 is also used to generate a system read-write scheduling task based on the data index information of the medical data; schedule the scheduling task according to the system read-write scheduling algorithm; when the scheduling task is scheduled, read the persistently stored medical data from the disk based on the scheduling task.

[0180] Each module in the above-mentioned medical data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0181] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store medical data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a medical data processing method is implemented.

[0182] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0184] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0185] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A medical data processing method, characterized in that: The method comprises: Get the current data offset of the corresponding medical data from the message queue; Obtaining data index information corresponding to the current data offset; Based on the data index information, reading the persistently stored medical data; The read persistently stored medical data is sent to the data recipient.

2. The method according to claim 1, characterized in that The method further comprises: Receiving medical data to be processed sent by the data sender; Persistently storing the medical data to be processed in an asynchronous manner; Obtaining the data offset of the persistent storage of the medical data to be processed, and storing the data offset in the corresponding message queue; Based on the attributes of the medical data to be processed, data index information of the medical data to be processed is calculated, and the data index information is stored.

3. The method according to claim 2, characterized in that Before the to-be-processed medical data is persistently stored, the method further includes: Open a transaction corresponding to the medical data to be processed.

4. The method according to claim 3, characterized in that After the to-be-processed medical data is persistently stored, the method further includes: When the pending medical data is successfully persistently stored, submitting the transaction to the data sender; When the persistent storage of the to-be-processed medical data fails, transaction rollback information is sent to the data sender.

5. The method according to claim 2, characterized in that: After storing the data index information, the method further includes: Back up the persistently stored medical data, message queues, and data index information to a backup server.

6. The method according to claim 1, characterized in that The step of obtaining the current data offset of the medical data corresponding to the data receiver from the message queue includes: Obtain the topic corresponding to the data recipient, and determine the message queue corresponding to the topic; The current data offset of the medical data corresponding to the data receiver is obtained from the determined message queue.

7. The method according to claim 1, characterized in that After sending the read persistently stored medical data to the data receiver, the method includes: Updates the current data offset in the message queue.

8. The method according to any one of claims 1 to 7, characterized in that: The step of obtaining the current data offset of the medical data corresponding to the data receiver from the message queue includes: Obtain the current data offset of the medical data corresponding to the data receiver from the message queue stored in the memory; The acquiring of data index information corresponding to the current data offset includes: The data index information corresponding to the current data offset is obtained from the index information file stored in the memory.

9. The method according to any one of claims 1 to 7, characterized in that: The step of reading the persistently stored medical data based on the data index information includes: Based on the data index information, medical data is read from the memory, where the medical data stored in the memory is obtained by processing the medical data persistently stored in the disk based on the memory mapping method; When the medical data is not read from the memory, the persistently stored medical data is read from the disk based on the data index information.

10. The method according to claim 9, characterized in that The step of reading the persistently stored medical data from the disk includes: Generate system read and write scheduling tasks based on data index information of the medical data; Scheduling the scheduling task according to the system read-write scheduling algorithm; When the scheduling task is scheduled, the persistently stored medical data is read from the disk based on the scheduling task.

11. A medical data processing device, characterized in that: The device comprises: A current data offset acquisition module is used to obtain the current data offset of the medical data corresponding to the data receiver from the message queue; An index information acquisition module, used to acquire data index information corresponding to the current data offset; A data reading module, used for reading the persistently stored medical data based on the data index information; The consumption module is used to send the read persistently stored medical data to the data recipient.

12. The device according to claim 11, characterized in that The device also includes: A data receiving module, used for receiving the medical data to be processed sent by the data sender; A persistent storage module, used for persistently storing the medical data to be processed in an asynchronous manner; A message queue storage module, used for obtaining the data offset of the persistent storage of the medical data to be processed, and storing the data offset in the corresponding message queue; The index information storage module is used to calculate the data index information of the medical data to be processed based on the attributes of the medical data to be processed, and store the data index information.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images