Message peak clipping processing method based on efficient caching and batch operation

Abstract

The invention belongs to the field of data processing, and particularly relates to a message peak clipping processing method based on efficient caching and batch operation, which comprises the following steps of: S1, jointly forming a message peak clipping processing framework through a medical message generation layer, a Redis List caching layer, a Java batch processing layer and a MySQL persistence layer; s2, enqueueing the medical data information; and S3, a message triggering module of the Java batch processing layer performs batch processing triggering by adopting double mechanisms of timing triggering and quantity threshold triggering. According to the method, the Redis List is used as an efficient cache carrier of the medical data messages, batch pulling, batch processing and batch persistence of the medical data messages are achieved in combination with the Java technology, the problems of high-concurrency peak clipping and efficient data processing in medical data docking interaction are solved, and safe and timely circulation of medical data such as patient information, hospitalization and discharge information and medical advice is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a message smoothing method based on efficient caching and batch operations. Background Technology

[0002] In the current development of medical informatization, various hospital business systems (such as HIS hospital information system, LIS laboratory information system, PACS image archiving and communication system, etc.) need to frequently exchange and connect with medical data such as patient-related information (such as basic identity information, medical history records), inpatient and discharge information (such as admission registration, discharge settlement data), patient medical orders (such as medication instructions, examination requests), staff information (such as medical staff job information, shift data), and hospital department and ward bed information (such as department distribution, ward bed occupancy status).

[0003] In existing technologies, medical data exchange and interaction often adopt the method of "business systems directly writing to / reading from the database": when a business system generates medical data (such as generating hospitalization information when a patient is admitted), the system directly calls the database interface to write the data to a relational database such as MySQL; other business systems that depend on this data (such as the nurse station system that needs to obtain hospitalization information to arrange beds) directly read the data from the database.

[0004] However, this approach has significant drawbacks during peak medical service periods (such as the morning rush hour for patient admission registration, concentrated discharge settlement, or when hospital systems are synchronizing historical medical data in batches): (1) Database pressure overload: During peak periods of medical data connection and interaction, hundreds or even thousands of data write / read requests may be generated per second. A large number of single SQL operations directly flood the MySQL database, causing the database connection pool to be exhausted and the SQL execution queue to accumulate. In severe cases, this can cause the database to crash, affecting the normal operation of medical services. (2) High data processing latency: Writing / reading a single piece of data requires a complete process such as establishing a database connection, executing SQL, and releasing the connection. In high-concurrency scenarios, the processing time for a single piece of data increases from milliseconds to seconds, causing downstream business systems that depend on the data to be unable to obtain the latest data in a timely manner, resulting in data asynchrony problems such as "the patient has been admitted but the bed occupancy status has not been updated"; (3) High risk of data loss: If the business system encounters network fluctuations or temporary database failures when writing directly to the database, the data that has not been written will be lost directly. Furthermore, there is a lack of effective data storage and compensation mechanisms. Medical data is critical data, and its loss will seriously affect the quality of medical services and may even lead to medical safety hazards. Based on the above shortcomings, we propose a message smoothing method based on efficient caching and batch operations. Summary of the Invention

[0005] This invention proposes a message smoothing processing method based on efficient caching and batch operations, which solves the problems existing in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A message smoothing method based on efficient caching and batch operations includes the following steps: S1: A message smoothing architecture is formed by a medical message generation layer, a Redis List caching layer, a Java batch processing layer, and a MySQL persistence layer. The medical message generation layer consists of the HIS system, nurse station system, medical order system, employee management system, and department bed management system. The Redis List caching layer is used to deploy Redis services and create a Redis List data structure named dic:data:message:queue as a temporary cache queue for medical data messages. The Java batch processing layer is developed and deployed on the medical data integration and interaction platform based on Java. Its core components include a message triggering module, a batch retrieval module, a message parsing and business processing module, a batch persistence module, and an exception compensation module. The MySQL persistence layer deploys a MySQL database, which contains medical data-related data tables. S2: Medical data information is enqueued, and the specific steps are as follows: S201: The medical message generation layer, as the source of medical data messages, is responsible for encapsulating medical data such as patient information, hospitalization and discharge information, medical orders, employee information, and department bed information into standardized messages. It writes the messages to the Redis List cache layer by calling the LPUSH command through the Java Redis client and enqueues them from the left side of the Redis List cache layer. S202: The Redis List cache layer automatically sorts messages according to their enqueue order, enabling ordered temporary storage of messages; S3: The message triggering module of the Java batch processing layer uses a dual mechanism of "timed triggering + quantity threshold triggering" for batch processing triggering. The specific steps are as follows: S301: Scheduled triggering creates scheduled tasks through Java ScheduledExecutorService, configures the trigger period, and schedules batch processing flows to be triggered at regular intervals; S302: Quantity threshold triggering uses the Java atomic class AtomicInteger to count the number of messages in the Redis List in real time. When the number of messages reaches a preset threshold, the batch processing process is immediately triggered. S303: When the timer period has not arrived but the number of messages has reached the threshold, batch processing triggered by the quantity threshold will be executed first to avoid message backlog. When the timer period has arrived but the number of messages has not reached the threshold, batch processing will still be executed to ensure that a small number of messages can be processed in a timely manner. S4: After the batch processing flow is triggered, the processing steps of the Java batch processing layer include the following steps: S401: The batch retrieval module of the Java batch processing layer batch retrieves the Redis LRANGE dic:data:message:queue 0 N-1 command, where N is the batch retrieval quantity, consistent with the message quantity threshold. It reads the first N medical data messages from the RedisList in batch; at the same time, it immediately calls the LTRIM dic:data:message:queue N-1 command to delete the first N messages already read from the RedisList, ensuring that these messages are not repeatedly retrieved by other processing threads. S402: The message parsing and business processing module of the Java batch processing layer performs streaming processing on batch-fetched messages through the Java Stream API. The specific process is as follows: S4021: During message parsing, the message parsing and business processing modules parse the message according to different message formats and convert the message into the corresponding Java entity class object; S4022: During business verification, the message parsing and business processing module executes exclusive verification logic for different types of medical data messages. It verifies whether the message content already exists based on the unique data identifier, filters invalid or illegal messages, and non-empty fields cannot be empty. S4023: During data conversion, the message parsing and business processing module converts the validated objects into the VO entity class corresponding to the MySQL data table, in preparation for subsequent persistence; S403: The batch persistence module of the Java batch processing layer uses MyBatis's batch insert interface to write the converted VO entity classes into the corresponding MySQL tables in batches. For different types of medical data messages, the corresponding batch insert method is called. The SQL statement for batch insert is: INSERT INTO t_sick_info (sick_no, sick_serial_no, sick_name, sick_in_time, dept_code, area_code, bed_code) VALUES(?, ?, ?, ?, ?, ?, ?, ?), where multiple sets of parameters after VALUES enable batch writing. S404: When the exception compensation module of the Java batch processing layer detects an exception during the batch persistence process, it immediately triggers a transaction rollback, cancels the executed batch write operation, rewrites the medical data messages fetched in this batch to the Redis standby list, adds a retry count flag to the message, and then starts a compensation scheduled task to read messages with ≤3 retries from the standby list and push them back to the standby list for reprocessing. If the number of retries exceeds 3, the message is written to the MySQL exception data table. S5: The MySQL persistence layer receives batch write requests from the Java batch processing layer to achieve persistent storage of medical data and provide data query services for downstream business systems.

[0007] Preferably, in step S202, the Redis service of the Redis List cache layer enables AOF persistence and appends LPUSH operations to the AOF log file in real time to avoid the loss of temporarily stored medical data messages when the Redis service crashes abnormally.

[0008] Preferably, in S301, the triggering cycle of the timed trigger can be dynamically adjusted according to the peak period of medical business.

[0009] Preferably, in step S403, a MySQL transaction is started during batch writing to ensure that all medical data messages in the same batch are either written successfully or rolled back completely, avoiding data inconsistency caused by "partial message writing success and partial failure".

[0010] Preferably, in S404, the name of the spare list is dic:data:message:fail:queue.

[0011] Preferably, in S1, the medical data related data tables include t_sick_info patient information table, t_sick_out_info inpatient discharge table, t_sick_medical_order medical order table, t_dept_info department table, t_area_info ward table, and t_bed_info bed table.

[0012] Compared with existing technologies, this invention uses Redis List as an efficient caching carrier for medical data messages, and combines Java technology to realize batch retrieval, batch processing and batch persistence of medical data messages. It solves the problems of high concurrency peak shaving and efficient data processing in medical data docking and interaction, and ensures the safe and timely flow of medical data such as patient information, hospitalization and discharge information, and medical orders. Attached Figure Description

[0013] Figure 1This is a connection block diagram of a message peak shaping processing method based on efficient caching and batch operations proposed in this invention; Figure 2 This is a flowchart of a message peak shaping method based on efficient caching and batch operations proposed in this invention; Figure 3 for Figure 2 A flowchart for batch processing of traditional Chinese medicine data messages. Detailed Implementation

[0014] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0015] Reference Figure 1-3 A message smoothing method based on efficient caching and batch operations includes the following steps: S1: A message smoothing architecture is formed by a medical message generation layer, a Redis List caching layer, a Java batch processing layer, and a MySQL persistence layer. The medical message generation layer consists of the HIS system, nurse station system, medical order system, employee management system, and department bed management system. The Redis List caching layer is used to deploy Redis services and create a Redis List data structure named dic:data:message:queue as a temporary cache queue for medical data messages. The Java batch processing layer is developed and deployed on the medical data integration and interaction platform based on Java. Its core components include a message triggering module, a batch retrieval module, a message parsing and business processing module, a batch persistence module, and an exception compensation module. The MySQL persistence layer deploys a MySQL database, which contains medical data-related data tables, such as t_sick_info patient information table, t_sick_out_info inpatient discharge table, t_sick_medical_order medical order table, t_dept_info department table, t_area_info ward table, and t_bed_info bed table. S2: Medical data information is enqueued, and the specific steps are as follows: S201: The medical message generation layer, as the source of medical data messages, is responsible for encapsulating medical data such as patient information, hospitalization and discharge information, medical orders, employee information, and department bed information into standardized messages. It writes the messages to the Redis List cache layer by calling the LPUSH command through the Java Redis client and enqueues them from the left side of the Redis List cache layer. S202: The Redis List cache layer automatically sorts messages according to their enqueue order, enabling ordered temporary storage of messages. The Redis service of the RedisList cache layer enables AOF persistence, appending LPUSH operations to the AOF log file in real time to prevent the loss of temporarily stored medical data messages when the Redis service crashes abnormally. S3: The message triggering module of the Java batch processing layer uses a dual mechanism of "timed triggering + quantity threshold triggering" for batch processing triggering. The specific steps are as follows: S301: Scheduled triggering creates scheduled tasks through Java ScheduledExecutorService, configures the trigger period, and the trigger period can be dynamically adjusted according to the peak period of medical business to trigger batch processing processes. S302: Quantity threshold triggering uses the Java atomic class AtomicInteger to count the number of messages in the Redis List in real time. When the number of messages reaches a preset threshold, the batch processing process is immediately triggered. S303: When the timer period has not arrived but the number of messages has reached the threshold, batch processing triggered by the quantity threshold will be executed first to avoid message backlog. When the timer period has arrived but the number of messages has not reached the threshold, batch processing will still be executed to ensure that a small number of messages can be processed in a timely manner. S4: After the batch processing flow is triggered, the processing steps of the Java batch processing layer include the following steps: S401: The batch retrieval module of the Java batch processing layer batch retrieves the Redis LRANGE dic:data:message:queue 0 N-1 command, where N is the batch retrieval quantity, consistent with the message quantity threshold. It reads the first N medical data messages from the RedisList in batch; at the same time, it immediately calls the LTRIM dic:data:message:queue N-1 command to delete the first N messages already read from the RedisList, ensuring that these messages are not repeatedly retrieved by other processing threads. S402: The message parsing and business processing module of the Java batch processing layer performs streaming processing on batch-fetched messages through the Java Stream API. The specific process is as follows: S4021: During message parsing, the message parsing and business processing modules parse the message according to different message formats and convert the message into the corresponding Java entity class object; S4022: During business verification, the message parsing and business processing module executes exclusive verification logic for different types of medical data messages. It verifies whether the message content already exists based on the unique data identifier, filters invalid or illegal messages, and non-empty fields cannot be empty. S4023: During data conversion, the message parsing and business processing module converts the validated objects into the VO entity class corresponding to the MySQL data table, in preparation for subsequent persistence; S403: The batch persistence module of the Java batch processing layer uses MyBatis's batch insert interface to write the converted VO entity classes into the corresponding MySQL tables in batches. For different types of medical data messages, the corresponding batch insert method is called. The SQL statement for batch insert is INSERT INTO t_sick_info (sick_no, sick_serial_no, sick_name, sick_in_time, dept_code, area_code, bed_code) VALUES(?, ?, ?, ?, ?, ?, ?, ?), where multiple sets of parameters after VALUES are used to achieve batch writing. During batch writing, a MySQL transaction is started to ensure that the medical data messages in the same batch are either all written successfully or all rolled back, avoiding data inconsistency caused by "partial message writing success and partial failure". S404: When the exception compensation module of the Java batch processing layer detects an exception during the batch persistence process, it immediately triggers a transaction rollback, cancels the executed batch write operations, and rewrites the medical data messages fetched in this batch into the Redis standby list. It also adds a retry count flag to the messages, where the standby list is named dic:data:message:fail:queue. Then, it starts a compensation scheduled task to read messages with ≤3 retries from the standby list and push them back to the standby list for reprocessing. If the number of retries exceeds 3, the messages are written to the MySQL exception data table. S5: The MySQL persistence layer receives batch write requests from the Java batch processing layer to achieve persistent storage of medical data and provide data query services for downstream business systems; This embodiment uses Redis List as a high-efficiency caching carrier for medical data messages, and combines Java technology to realize batch retrieval, batch processing and batch persistence of medical data messages. It solves the problems of high concurrency peak shaving and efficient data processing in medical data docking and interaction, and ensures the safe and timely flow of medical data such as patient information, hospitalization and discharge information, and medical orders.

[0016] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A message peak shaping method based on efficient caching and batch operations, characterized in that, Includes the following steps: S1: A message smoothing architecture is formed by the medical message generation layer, Redis List caching layer, Java batch processing layer, and MySQL persistence layer. The medical message generation layer consists of the HIS system, nurse station system, medical order system, employee management system, and department bed management system. The Redis List caching layer is used to deploy Redis services and create a Redis List data structure named dic:data:message:queue as a temporary cache queue for medical data messages. The Java batch processing layer is developed and deployed on the medical data integration and interaction platform based on Java. Its core components include a message triggering module, a batch retrieval module, a message parsing and business processing module, a batch persistence module, and an exception compensation module. The MySQL persistence layer deploys a MySQL database, which contains data tables related to medical data. S2: Medical data information is enqueued, and the specific steps are as follows: S201: The medical message generation layer, as the source of medical data messages, is responsible for encapsulating medical data such as patient information, hospitalization and discharge information, medical orders, employee information, and department bed information into standardized messages. It writes the messages to the Redis List cache layer by calling the LPUSH command through the Java Redis client and enqueues them from the left side of the Redis List cache layer. S202: The Redis List cache layer automatically sorts messages according to their enqueue order, enabling ordered temporary storage of messages; S3: The message triggering module of the Java batch processing layer uses a dual mechanism of "timed triggering + quantity threshold triggering" for batch processing triggering. The specific steps are as follows: S301: Scheduled triggering creates scheduled tasks through Java ScheduledExecutorService, configures the trigger period, and schedules batch processing flows to be triggered at regular intervals; S302: Quantity threshold triggering uses the Java atomic class AtomicInteger to count the number of messages in the Redis List in real time. When the number of messages reaches a preset threshold, the batch processing process is immediately triggered. S303: When the timer period has not arrived but the number of messages has reached the threshold, batch processing triggered by the quantity threshold will be executed first to avoid message backlog. When the timer period has arrived but the number of messages has not reached the threshold, batch processing will still be executed to ensure that a small number of messages can be processed in a timely manner. S4: After the batch processing flow is triggered, the processing steps of the Java batch processing layer include the following steps: S401: The batch retrieval module of the Java batch processing layer batch retrieves the Redis LRANGE dic:data:message:queue 0 N-1 command, where N is the batch retrieval quantity, consistent with the message quantity threshold. It reads the first N medical data messages from the Redis List in batches; at the same time, it immediately calls the LTRIM dic:data:message:queue N -1 command to delete the first N messages already read from the Redis List, ensuring that these messages are not repeatedly retrieved by other processing threads. S402: The message parsing and business processing module of the Java batch processing layer performs streaming processing on batch-fetched messages through the Java Stream API. The specific process is as follows: S4021: During message parsing, the message parsing and business processing modules parse the message according to different message formats and convert the message into the corresponding Java entity class object; S4022: During business verification, the message parsing and business processing module executes exclusive verification logic for different types of medical data messages. It verifies whether the message content already exists based on the unique data identifier, filters invalid or illegal messages, and non-empty fields cannot be empty. S4023: During data conversion, the message parsing and business processing module converts the validated objects into the VO entity class corresponding to the MySQL data table to prepare for subsequent persistence; S403: The batch persistence module of the Java batch processing layer uses MyBatis's batch insert interface to write the converted VO entity classes into the corresponding MySQL tables in batches. For different types of medical data messages, the corresponding batch insert method is called. The SQL statement for batch insert is: INSERT INTO t_sick_info (sick_no, sick_serial_no, sick_name, sick_in_time, dept_code, area_code, bed_code) VALUES(?, ?, ?, ?, ?, ?, ?, ?), where multiple sets of parameters after VALUES enable batch writing. S404: When the exception compensation module of the Java batch processing layer detects an exception during the batch persistence process, it immediately triggers a transaction rollback, cancels the executed batch write operation, rewrites the medical data messages fetched in this batch to the Redis standby list, adds a retry count flag to the message, and then starts a compensation scheduled task to read messages with ≤3 retries from the standby list and push them back to the standby list for reprocessing. If the number of retries exceeds 3, the message is written to the MySQL exception data table. S5: The MySQL persistence layer receives batch write requests from the Java batch processing layer to achieve persistent storage of medical data and provide data query services for downstream business systems.

2. The message peak shaping method based on efficient caching and batch operations according to claim 1, characterized in that, In step S202, the Redis service of the Redis List cache layer enables AOF persistence and appends LPUSH operations to the AOF log file in real time to prevent the loss of temporarily stored medical data messages when the Redis service crashes abnormally.

3. The message peak shaping method based on efficient caching and batch operations according to claim 1, characterized in that, In S301, the triggering cycle of timed triggers can be dynamically adjusted according to the peak periods of medical business.

4. The message peak shaping method based on efficient caching and batch operations according to claim 1, characterized in that, In S403, MySQL transactions are enabled during batch writing to ensure that medical data messages in the same batch are either all written successfully or all rolled back, avoiding data inconsistency caused by "partial message writing success and partial failure".

5. The message peak shaping method based on efficient caching and batch operations according to claim 1, characterized in that, In S404, the name of the backup List is dic:data:message:fail:queue.

6. The message peak shaping method based on efficient caching and batch operations according to claim 1, characterized in that, In S1, medical data-related tables include t_sick_info patient information table, t_sick_out_info inpatient discharge table, t_sick_medical_order medical order table, t_dept_info department table, t_area_info ward table, and t_bed_info bed table.

Images