Distributed log processing method and system based on message driving

Abstract

The invention provides a distributed log processing method and system based on message driving, and relates to the field of medical informatization. The method comprises the following steps: asynchronously collecting multi-source original log data, packaging the multi-source original log data into a log event message, and delivering the log event message to a distributed message queue through a non-blocking interface; and distributing the log event messages to corresponding message queues by utilizing a preset routing rule according to the metadata features of the log event messages. Performing parallel consumption processing on the log event messages in the message queues by utilizing an Actor model cluster; wherein during parallel consumption processing, the number of instances of an Actor cluster is dynamically adjusted according to a threshold value by monitoring the partition backlog amount of a message queue and the load rate of the Actor model cluster, the running state of the Actor model cluster is monitored through a supervisor hierarchical structure, restarting or message rerouting processing is conducted on a faulty Actor model, and the number of instances of the Actor cluster is reduced. And processing the failure message in combination with the dead message queue. The scheme can be matched with the core requirements of high sensitivity, high compliance and high real-time performance of medical applications.

Description

Technical Field

[0001] This application relates to the field of medical informatics, and more specifically, to a message-driven distributed log processing method and system. Background Technology

[0002] In the field of healthcare informatics, various diagnostic and treatment devices, hospital information systems (HIS), laboratory systems (LIS), picture archiving systems (PACS), and electronic medical records (EMR) generate massive amounts of operational and business logs every moment. These logs are crucial information carriers that record patient treatment processes, equipment operating status, system access behavior, and abnormal security events.

[0003] Currently, most traditional log processing solutions in the medical field adopt a centralized architecture. Typical implementations include: medical applications recording logs synchronously (e.g., directly writing to local files or databases); using a log collection agent to push log files to a centralized storage system; and finally, offline analysis and querying through batch jobs. These solutions typically rely on fixed log formats and predefined processing flows, and use locking mechanisms to resolve concurrency conflicts during multi-threaded writes. In other words, this traditional log system, with its synchronous writing and tightly coupled centralized architecture, suffers from inherent flaws such as performance bottlenecks, poor scalability, weak real-time performance, insufficient flexibility, and low reliability when facing high-concurrency scenarios, making it difficult to meet the core requirements of medical logs for high sensitivity, strong compliance, and high real-time performance. Summary of the Invention

[0004] The purpose of this application is to provide a message-driven distributed log processing method and system that can meet the core requirements of medical logs, which are highly sensitive, compliant, and real-time.

[0005] This application is implemented as follows:

[0006] Firstly, this application provides a message-driven distributed log processing method, comprising the following steps: asynchronously collecting multi-source raw log data, encapsulating it into log event messages, and delivering them to a distributed message queue through a non-blocking interface to obtain log event messages stored in the queue. Based on the metadata characteristics of the log event messages, the log event messages are allocated to the corresponding message queues using preset routing rules. An Actor model cluster is used to consume the log event messages in each message queue in parallel to obtain structured log data. During parallel consumption, the number of Actor cluster instances is dynamically adjusted according to a threshold by monitoring the partition backlog of the message queue and the load rate of the Actor model cluster. The running status of the Actor model cluster is monitored through a supervisor hierarchy, and faulty Actor models are restarted or rerouted. Failure messages are also processed using a dead-letter queue.

[0007] In some implementations, the asynchronous collection of multi-source raw log data and its encapsulation into log event messages includes: using lightweight edge nodes deployed at the log source end to filter, compress, and encrypt the raw log data to obtain a first log stream data; performing real-time log model recognition and classification on the first log stream data based on a pre-trained neural network model, wherein the neural network model is used to distinguish log types and attach semantic tags; serializing the log data stream with attached semantic tags into a columnar storage format log event message in user space, and writing it into a distributed message queue composed of remote persistent memory via the RDMA network protocol.

[0008] In some implementations, the metadata features include log type, source IP, and service priority.

[0009] In some implementations, the step of allocating log event messages to corresponding message queues based on the metadata characteristics of the log event messages using preset routing rules includes: calculating the hash value corresponding to the metadata characteristics of the log event messages according to a consistent hashing algorithm; and mapping the calculated hash value to the message queue partition range so that log event messages from the same device or user are always routed to the same fixed message queue partition.

[0010] In some implementations, the Actor model cluster includes multiple Actor processing units divided by function; the Actor processing unit subscribes to the corresponding partition queue of the message queue, asynchronously receives log event messages through the mailbox mechanism, and performs operations including log parsing, compliance verification, business rule execution, and storage preprocessing.

[0011] In some implementations, the parallel consumption and processing of log event messages in each message queue using an Actor model cluster includes: the Actor model cluster, which contains a parsing unit, a verification unit, a rule engine unit, and a storage unit, consumes and processes the log event messages in parallel, sequentially completing format parsing, compliance verification, rule matching, and persistence processing.

[0012] In some implementations, the step of dynamically adjusting the number of Actor cluster instances based on thresholds by monitoring the partition backlog of message queues and the load rate of the Actor model cluster includes: real-time monitoring of the partition backlog of each message queue and the load rate of the Actor model cluster. When the partition backlog of any message queue exceeds a first preset threshold, or the load rate of the Actor model cluster exceeds a second preset threshold, a new Actor model is added and assigned to the corresponding message queue; when the partition backlog of any message queue is lower than a third preset threshold, or the load rate of the Actor model cluster is lower than a fourth preset threshold, the corresponding idle Actor model is destroyed. The first preset threshold is greater than the third preset threshold, and the second preset threshold is greater than the fourth preset threshold.

[0013] In some implementations, the process of handling failed messages using a dead-letter queue includes: delivering messages that still fail to be processed after the maximum number of retries to a specific dead-letter queue. A monitoring service periodically polls the dead-letter queue, analyzes and records the messages, and triggers alarms to notify administrators to intervene manually or configure new processing rules for retries.

[0014] Secondly, this application provides a message-driven distributed log processing system, comprising: a data source layer, used for asynchronously collecting multi-source raw log data, encapsulating it into log event messages, and delivering them to a distributed message queue through a non-blocking interface to obtain log event messages stored in the queue; a message hub layer, used for allocating log event messages to corresponding message queues based on the metadata characteristics of the log event messages and using preset routing rules; and an intelligent processing layer, used for parallel consumption processing of log event messages in each message queue using an Actor model cluster to obtain structured log data; wherein, during parallel consumption processing, the number of Actor cluster instances is dynamically adjusted according to a threshold by monitoring the partition backlog of the message queue and the load rate of the Actor model cluster, and the running status of the Actor model cluster is monitored through a supervisor hierarchy structure, restarting or rerouting messages for faulty Actor models, and handling failed messages in conjunction with a dead-letter queue.

[0015] In some implementations, the system further includes: an analysis storage layer for scenario-based storage of log data and cold data archiving, including real-time storage components, offline storage components, and cold data archiving components; and an application presentation layer for outputting real-time analysis results, including real-time data dashboards, anomaly alarm components, and intelligent decision support components.

[0016] Thirdly, this application provides an electronic device including a memory for storing one or more programs; a processor; and, when the one or more programs are executed by the processor, implementing the method as described in any one of the first aspects above.

[0017] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0018] Fifthly, this application provides a computer program product including computer program instructions that, when executed by a processor, implement the method as described in any one of the first aspects above.

[0019] Compared with the prior art, this application has at least the following advantages or beneficial effects:

[0020] This application proposes a message-driven distributed log processing method. Based on the core processing concept of "asynchronous decoupling + distributed elastic processing," it constructs an asynchronous, decoupled, elastic, and highly fault-tolerant distributed log processing flow to meet the core requirements of high sensitivity, strong compliance, and high real-time performance in medical logs. Specifically, to address performance bottlenecks, asynchronous collection and non-blocking delivery are adopted, decoupling log production from medical business execution and fundamentally eliminating I / O blocking, resulting in ultra-high throughput and zero business interference. To address format fragmentation and compliance risks, metadata feature routing and Actor modular processing (such as independent verification units) achieve log format normalization and real-time anonymization, resulting in cross-system data unification and built-in compliance guarantees. To address poor real-time performance, streaming parallel processing based on the Actor model replaces batch jobs, achieving end-to-end millisecond-level latency and real-time event awareness. To address high coupling, the message-driven architecture abstracts processing logic into dynamically pluggable Actor components, resulting in significantly optimized system flexibility and maintenance costs. To address reliability issues, the supervisor hierarchy and dead-letter queue mechanism provide built-in fault self-healing capabilities, thereby achieving medical-grade reliability with high system availability and zero data loss. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of an embodiment of a message-driven distributed log processing method according to this application;

[0023] Figure 2 This is a structural block diagram of an embodiment of a message-driven distributed log processing system according to this application;

[0024] Figure 3 This is a structural block diagram of the storage layer and application presentation layer in one embodiment of a message-driven distributed log processing system according to this application;

[0025] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0026] Icons: 201, Processor; 202, Memory; 203, Communication Interface. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0028] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0029] Application Overview

[0030] In the wave of medical informatization and intelligentization, log data has become a core basis for medical equipment monitoring, electronic medical record traceability, and clinical decision support. However, when traditional log systems process massive, multi-source, and highly sensitive log data in medical scenarios, they expose key problems such as performance bottlenecks, insufficient real-time performance, and lack of compliance, which seriously restrict the reliability and response efficiency of medical systems.

[0031] Currently, most traditional log processing solutions in the medical field adopt a centralized architecture. These solutions typically rely on fixed log formats and predefined processing flows, and use locking mechanisms to resolve concurrency conflicts during multi-threaded writes. However, considering the specific needs of the medical field, these existing technologies have significant limitations: First, synchronous write mechanisms lead to performance bottlenecks. In high-concurrency diagnostic and treatment scenarios (such as peak registration, payment, or PACS image retrieval), frequent I / O waits can block core business threads, reduce system response speed, and even affect diagnostic and treatment efficiency and patient experience. Second, there are fragmented medical log formats and compliance risks. Logs generated by devices and systems from different vendors vary greatly in format, field definitions, and timestamps. This fragmentation makes cross-system transaction tracing and auditing extremely difficult. Furthermore, traditional systems lack real-time anonymization mechanisms for sensitive medical data (such as patient ID numbers and medical record content), posing a risk of violating data privacy regulations such as HIPAA and GDPR. Third, the lack of real-time processing capabilities fails to meet the needs of medical monitoring. Batch processing mode results in delays of several hours or even days between log generation and queryability, making it impossible for the system to detect critical events such as medical equipment malfunctions, suspected fraudulent registrations, or unauthorized access to sensitive medical records in real time, thus failing to provide immediate assurance for medical quality and safety. Fourth, high coupling leads to system rigidity, with log processing logic tightly intertwined with medical business code. If adjustments to the log audit format or content are needed to meet new regulations, it may require modifying a large amount of core business code and redeploying the system, resulting in extremely high maintenance costs and poor flexibility. Fifth, there are issues with resource contention and reliability. Traditional multi-threaded log writing relies on locking mechanisms, which are prone to performance drops or deadlocks under high pressure, potentially leading to the loss of critical medical logs. This is unacceptable in a medical scenario where data integrity is extremely important.

[0032] To address the aforementioned technical issues, this application provides a message-driven distributed log processing method. First, it decouples log production and consumption through asynchronous collection and non-blocking delivery, avoiding I / O wait times. Second, it introduces a distributed message queue to buffer and reliably transmit log data. Third, it employs an Actor model cluster for parallel consumption processing, where flexible scheduling is achieved through preset routing rules, and high availability and reliability are ensured through elastic scaling and a supervisor fault-tolerance mechanism. This method fundamentally solves the inherent defects of traditional solutions, achieving high-performance, highly scalable log processing that meets medical compliance requirements.

[0033] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0034] Exemplary methods

[0035] Please see Figure 1 This message-driven distributed log processing method includes the following steps: S1: Asynchronously collect multi-source raw log data, encapsulate it into log event messages, and deliver them to a distributed message queue through a non-blocking interface to obtain the log event messages stored in the queue. S2: Based on the metadata characteristics of the log event messages, allocate the log event messages to the corresponding message queues using preset routing rules. S3: Use an Actor model cluster to consume the log event messages in each message queue in parallel to obtain structured log data. During parallel consumption, the method includes: S4: Dynamically adjusting the number of Actor cluster instances according to thresholds by monitoring the partition backlog of the message queues and the load rate of the Actor model cluster; and S5: Monitoring the running status of the Actor model cluster through a supervisor hierarchy, restarting or rerouting messages for faulty Actor models, and handling failed messages using a dead-letter queue.

[0036] This application constructs an asynchronous, decoupled, elastic, and highly fault-tolerant distributed log processing flow to meet the core requirements of medical logs, which are highly sensitive, compliant, and real-time.

[0037] Specifically, firstly, a lightweight proxy or SDK can be used at the log source (such as medical devices or HIS systems) to directly capture raw log data. After capture, it is immediately encapsulated into "log event messages" in a predefined format (such as Protocol Buffers or Avro). Throughout this process, the producer calls a non-blocking interface (such as the asynchronous send API of Kafka Producer), sending the message and returning immediately without waiting for it to be actually written to the Broker node of the message queue. The message queue (such as Kafka) is responsible for receiving these messages and persisting them to disk, while providing a multi-replica mechanism to ensure no data loss. This provides a reliable data source for all subsequent processing. In essence, unstable, multi-source data streams are transformed into stable, uniformly formatted message streams, stored in a distributed message queue—a "buffer" and "hub." Non-blocking asynchronous writing completely liberates business threads, eliminating the need to wait for log I / O operations, greatly improving the throughput and response speed of the business system itself. Furthermore, the distributed message queue can handle sudden surges in traffic, preventing the backend processing system from being overwhelmed and improving the overall stability of the system. In addition, producers and consumers are separated in time and space, so a failure or restart on one side will not directly affect the other.

[0038] Next, after being stored in the distributed message queue, which serves as both a "buffer" and a "hub," the messages are not processed in an unordered manner. Instead, the metadata characteristics of the log event messages (such as log type, source IP, and business priority) are extracted, and intelligent distribution is performed according to preset routing rules (e.g., all high-priority logs from CT devices are routed to queue A). This step can use a rule engine and algorithms (such as consistent hashing) to classify and organize the chaotic message flow, directing it to different downstream processing channels, laying the foundation for parallel and differentiated processing. Specifically, a routing controller (such as a custom KafkaPartitioner or a standalone stream processor) can be used to consume the raw messages in the queue and extract their metadata characteristics (such as log type log_type = CT_Device, source IP ip = 192.168.1.10, and business priority priority = HIGH). Then, based on the preset routing rules (e.g., "all logs from CT devices are routed to the topic-log-ct queue" and "all high-priority logs are routed to the topic-priority-high queue"), the hash value corresponding to the metadata characteristics of the log event message is calculated using the consistent hashing algorithm. The calculated hash value is then mapped to the message queue partition range so that log event messages from the same device or user are always routed to the same fixed message queue partition.

[0039] It should be noted that by consistently routing logs from the same device or user to the same partition, the orderly processing of similar data can be guaranteed, facilitating time-series analysis and fully utilizing parallel processing capabilities. Furthermore, allocating independent queues and dedicated processing resources to high-priority messages (such as device alarms) ensures that critical business operations receive immediate responses, meeting the real-time requirements of healthcare.

[0040] Next, the log event messages after routing and classification are consumed and processed in parallel by the Actor model cluster. Each Actor model is an independent processing unit, which uses a mailbox mechanism to receive messages and processes them serially in a single thread, naturally avoiding concurrency lock issues. The processing flow can be modularly designed as a series of Actor models: (1) Parsing unit: responsible for deserializing messages, extracting key fields (such as extracting patient_id from JSON), and converting unstructured / semi-structured data into structured data. (2) Verification unit: responsible for verifying data integrity (such as whether required fields exist) and compliance (such as data anonymization of patient_name). (3) Rule engine unit: responsible for executing business rules (such as "if the same device reports 3 consecutive errors within 1 minute, an alarm will be triggered"). (4) Storage unit: responsible for writing the final processing results into the most suitable storage system (such as ES for retrieval and ClickHouse for analysis).

[0041] That is, based on the aforementioned scheme, in some implementations of this application, the Actor model cluster includes multiple Actor processing units divided by function; the Actor processing unit subscribes to the corresponding partition queue of the message queue, asynchronously receives log event messages through the mailbox mechanism, and performs operations including log parsing, compliance verification, business rule execution, and storage preprocessing.

[0042] For example, using an Actor model cluster to consume log event messages in each message queue in parallel can include: the Actor model cluster, which includes a parsing unit, a verification unit, a rule engine unit, and a storage unit, consumes and processes log event messages in parallel, sequentially completing format parsing, compliance verification, rule matching, and persistence processing.

[0043] Specifically, the Actor model cluster can be broken down into four dedicated processing units, each encapsulating independent core logic: the log parsing unit is responsible for converting unstructured logs (such as binary logs from medical devices and non-standard format maintenance logs) into structured data containing unified fields (timestamp, device ID, and log content); the compliance verification unit performs sensitive data anonymization (such as encrypting patient mobile phone numbers and medical record numbers) and industry compliance checks (such as medical HIPAA rule verification); the business rule execution unit processes data according to scenario-based rules (such as identifying equipment fault thresholds in medical scenarios and monitoring abnormal transactions in financial scenarios); and the storage preprocessing unit organizes the processed data according to storage specifications (such as sharding fields and indexing rules) to prepare for subsequent persistence.

[0044] Each Actor processing unit is bound to a message queue partition through a "partition subscription" mechanism. For example, the queue partition corresponding to medical device logs is subscribed to only by the medical-specific parsing unit, and the financial transaction log partition is subscribed to only by the financial rule execution unit. This ensures that log data is accurately routed to the corresponding processing unit and avoids cross-scenario data interference. Simultaneously, units communicate asynchronously via an "email mechanism": after the preceding unit (such as the parsing unit) completes its processing, it encapsulates the data into a message and delivers it to the dedicated email address of the following unit (such as the compliance verification unit). The following unit processes the data in a "first-in, first-out" order, without waiting for an immediate response from the preceding unit.

[0045] Continue reading Figure 1When using an Actor model cluster to consume log event messages in various message queues in parallel, to ensure resource utilization and processing efficiency, a monitoring system can continuously track the partition backlog of message queues (such as Kafka's Lag) and the load rate of the Actor cluster (such as CPU utilization). These real-time metrics are sent to a resource scheduler (such as Kubernetes' HPA). The scheduler dynamically adjusts the number of instances in the Actor model cluster by calling the container platform API, based on a preset threshold strategy (such as "expand if average Lag > 1000"). During expansion, new instances automatically register and join consumer groups to share the load; during downsizing, instances are shut down, and their assigned partitions are reassigned to the remaining instances. This process requires no manual intervention, reducing operational complexity and achieving adaptive matching of computing resources and workload, ensuring that processing capacity always matches the workload. This allows for automatic scaling based on real-time traffic, easily handling peak and off-peak periods in healthcare operations (such as morning appointment peaks), achieving true "on-demand usage," and saving costs.

[0046] Meanwhile, the supervisor hierarchy monitors the health of all working Actor models. If an Actor model fails, its supervisor immediately intervenes according to a predefined strategy (such as "restart") to automatically isolate and recover from the failure. Messages that fail after multiple retries are moved to a dead-letter queue, thus not affecting the processing of mainstream messages and awaiting subsequent manual investigation, ensuring eventual data consistency.

[0047] In summary, the above embodiments, based on the core processing concept of "asynchronous decoupling + distributed elastic processing," construct an asynchronous, decoupled, elastic, and highly fault-tolerant distributed log processing flow to meet the core requirements of high sensitivity, strong compliance, and high real-time performance of medical logs. Specifically, to address performance bottlenecks, asynchronous collection and non-blocking delivery are adopted, decoupling log production from medical business execution and fundamentally eliminating I / O blocking, resulting in ultra-high throughput and zero business interference. To address format fragmentation and compliance risks, metadata feature routing and Actor modular processing (such as independent verification units) achieve log format normalization and real-time anonymization, resulting in cross-system data unification and built-in compliance protection. To address poor real-time performance, streaming parallel processing based on the Actor model replaces batch jobs, achieving end-to-end millisecond-level latency and real-time event awareness. To address high coupling, the message-driven architecture abstracts processing logic into dynamically pluggable Actor components, resulting in significantly optimized system flexibility and maintenance costs. To address reliability issues, the supervisor hierarchy and dead-letter queue mechanism provide built-in fault self-healing capabilities, thereby achieving medical-grade reliability with high system availability and zero data loss.

[0048] Based on the aforementioned scheme, in some implementations of this application, asynchronous collection of multi-source raw log data and encapsulation into log event messages include: using lightweight edge nodes deployed at the log source end to filter, compress, and encrypt the raw log data to obtain first log stream data; performing real-time log model recognition and classification on the first log stream data based on a pre-trained neural network model, wherein the neural network model is used to distinguish log types and attach semantic tags; serializing the log data stream with attached semantic tags into log event messages in columnar storage format in user space, and writing them into a distributed message queue composed of remote persistent memory via the RDMA network protocol.

[0049] In the above implementation, by deploying lightweight edge nodes (CPU usage < 5%, memory usage < 100MB) at the log source end such as medical equipment and HIS system, the original logs are first processed by a three-step process of "filtering-compression-encryption": (1) filtering empty logs and duplicate logs to reduce invalid data; (2) using the ZIP compression algorithm (compression rate ≥ 60%) to reduce transmission bandwidth usage; (3) using AES-256 encryption to protect sensitive medical data (such as patient medical record logs) to generate "first log stream data" and avoid resource waste and data leakage risks at the source end.

[0050] Next, a pre-trained neural network model based on BERT can be used to perform real-time semantic analysis on the first log stream data. The model learns log features in the medical field (such as keywords like "equipment fault code" and "prescription number"), automatically distinguishes log types (such as CT equipment operation logs and electronic prescription operation logs), and adds semantic tags (such as "equipment malfunction" and "high-priority treatment") to each log.

[0051] Then, in user space, the log data stream with added semantic tags is serialized into a columnar storage format (such as Parquet), which can significantly reduce storage space compared to the row format (by about 40%). Through the RDMA (Remote Direct Memory Access) network protocol, log event messages are written directly to a distributed message queue composed of remote persistent memory, bypassing the operating system kernel. The delivery latency can be reduced from 10ms in traditional TCP / IP to less than 1ms, without consuming CPU resources at the log source end.

[0052] In summary, lightweight edge node preprocessing avoids excessive resource consumption at the source end. For example, when a surgical monitor is running, edge node log collection does not affect its core vital sign monitoring function. Compression processing significantly reduces network bandwidth consumption, avoiding network congestion caused by multiple devices simultaneously transmitting logs. Furthermore, traditional logs are raw text that requires post-parsing, while in the above implementation, a neural network model performs real-time analysis at the source, adding semantic tags to each log entry. This transforms logs into self-describing, structured intelligent data at the time of generation. Downstream systems do not need complex text parsing; they can directly route, analyze, and process based on semantic tags, greatly improving efficiency and laying a solid foundation for solving the problem of log format fragmentation.

[0053] Based on the aforementioned scheme, in some implementations of this application, the number of instances in the Actor cluster is dynamically adjusted according to thresholds by monitoring the partition backlog of message queues and the load rate of the Actor model cluster. This includes: real-time monitoring of the partition backlog of each message queue and the load rate of the Actor model cluster. When the partition backlog of any message queue exceeds a first preset threshold, or the load rate of the Actor model cluster exceeds a second preset threshold, a new Actor model is added and assigned to the corresponding message queue; when the partition backlog of any message queue is lower than a third preset threshold, or the load rate of the Actor model cluster is lower than a fourth preset threshold, the corresponding idle Actor model is destroyed. The first preset threshold is greater than the third preset threshold, and the second preset threshold is greater than the fourth preset threshold.

[0054] In the above implementation, four preset thresholds form control boundaries (first preset threshold > third preset threshold, second preset threshold > fourth preset threshold). This allows for dual-dimensional monitoring and threshold control, automatically scaling up during peak medical periods (such as morning registration peaks) to ensure log backlog does not exceed the threshold, and automatically scaling down during off-peak periods, solving the problem of "insufficient capacity during peak hours and wasted capacity during off-peak hours" in traditional fixed clusters. Furthermore, this control boundary design creates a "buffer zone," preventing frequent scaling up and down (scaling interval ≥ 5 minutes) caused by small fluctuations in log volume or load rate, reducing system resource jitter, and improving the stability of critical medical log processing.

[0055] For example, when the backlog of a partition exceeds the first threshold (e.g., 5000 records) or the Actor load rate exceeds the second threshold (e.g., 80%), a scaling-up mechanism can be triggered to add Actor instances according to "partition correspondence" (e.g., if there is a backlog of device log partitions, a new device log parsing Actor can be added); when the backlog is lower than the third threshold (e.g., 1000 records) and the load rate is lower than the fourth threshold (e.g., 30%), a scaling-down mechanism can be triggered to prioritize the destruction of Actor instances that have been idle for more than 5 minutes.

[0056] Based on the aforementioned scheme, some implementations in this application incorporate dead-letter queues to handle failed messages, including: delivering messages that still fail to be processed after the maximum number of retries to a specific dead-letter queue. A monitoring service periodically polls the dead-letter queue, analyzes and records the messages, and triggers alarms to notify administrators to intervene manually or configure new processing rules for retries.

[0057] In the above implementation, during the Actor cluster processing, messages that fail due to log format errors (such as missing fields in medical device logs) or dependent service failures (such as storage system timeouts) first execute a preset retry strategy (e.g., retrying 3 times at 100ms, 500ms, and 1s intervals). If the message still fails after the maximum number of retries, the system automatically marks it as a "dead letter" and delivers it to a dedicated dead letter queue (e.g., "device failure log dead letter queue" and "prescription operation log dead letter queue" in the medical scenario) according to "failure type + business priority" to avoid mixing with normal messages. Simultaneously, by deploying an independent monitoring service, each dead letter queue is polled every 30 seconds, collecting core information of the dead letter messages (failure reason, original log content, processing trajectory, business priority) and storing it in the log audit database. At the same time, a rule engine automatically analyzes failure patterns (e.g., failures due to frequent format updates in certain types of medical device logs) and generates a statistical report on failure reasons (e.g., "format errors account for 60%, dependent failures account for 30%)." Therefore, when the number of dead-letter messages in the dead-letter queue exceeds a preset threshold (e.g., more than 10 medical P0-level dead letters), or when the proportion of a specific failure type exceeds 50%, the monitoring service triggers multi-level alarms (SMS + email + system pop-up) to notify the administrator to intervene. The administrator can view the details of the dead letters through the visual interface. If the failure is caused by missing rules, new processing rules can be configured directly (e.g., adding a rule for parsing the medical device log format). The system will automatically send the rule to the Actor cluster and trigger the redelivery and processing of the corresponding messages in the dead-letter queue.

[0058] Exemplary System

[0059] Please see Figure 2 This application provides a message-driven distributed log processing system, which includes:

[0060] The data source layer asynchronously collects raw log data from multiple sources, encapsulates it into log event messages, and delivers them to a distributed message queue via a non-blocking interface, resulting in queue-stored log event messages. The message hub layer allocates log event messages to corresponding message queues based on their metadata characteristics and using preset routing rules. The intelligent processing layer utilizes an Actor model cluster to process log event messages from each message queue in parallel, resulting in structured log data. During parallel processing, it monitors the partition backlog of message queues and the load rate of the Actor model cluster, dynamically adjusting the number of Actor cluster instances according to thresholds. A supervisor hierarchy monitors the Actor model cluster's running status, restarting or rerouting faulty Actor models, and handling failed messages using a dead-letter queue.

[0061] For the specific implementation process of the above system, please refer to the message-driven distributed log processing method provided in the "Exemplary Methods" section, which will not be repeated here.

[0062] Please see Figure 3 Based on the aforementioned scheme, in some implementations of this application, the message-driven distributed log processing system further includes: an analysis storage layer for scenario-based storage of log data and cold data archiving, including real-time storage components, offline storage components, and cold data archiving components; and an application presentation layer for outputting real-time analysis results, including real-time data dashboards, anomaly alarm components, and intelligent decision support components.

[0063] In the above implementation, an analysis and storage layer and an application presentation layer are added to the original three-tier architecture (data source layer, message hub layer, and intelligent processing layer), forming a complete log processing closed loop. This achieves a full-link value loop for medical logs from "processing-storage-analysis-decision making," providing data-driven technical support for smart healthcare. Specifically, the real-time storage component can use an in-memory database (such as Redis) to store nearly 24 hours of log data, supporting high-frequency queries (such as real-time monitoring of surgical equipment status). The offline storage component can store historical logs based on a distributed file system (such as HDFS) or object storage (such as AWS S3), supporting batch analysis (such as patient treatment behavior pattern mining). The cold data archiving component can encrypt logs older than 30 days according to compliance requirements (such as HIPAA) and migrate them to low-cost storage (such as AWS Glacier) to meet long-term medical data retention regulations. The real-time data dashboard can push real-time log analysis results (such as surgical equipment failure rate and peak registration traffic) via WebSocket, supporting dynamic monitoring by medical administrators. The anomaly alert component can trigger multi-level alerts (SMS, email, HIS system pop-ups) for key indicators (such as a sudden drop in patient blood pressure) based on rule engines (such as Drools) and machine learning models (such as LSTM to predict equipment failure). The intelligent decision support component can integrate historical log data and external knowledge bases (such as medical guidelines) and use graph neural networks (GNNs) to analyze the compliance of treatment pathways, providing doctors with auxiliary decision-making suggestions.

[0064] Exemplary electronic devices

[0065] Please see Figure 4 This application provides an electronic device including at least one processor 201 and at least one memory 202. The processor 201 and memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction. The memory 202 stores program instructions executable by the processor 201, which can call the program instructions to execute them, thereby implementing a message-driven distributed log processing method according to various embodiments of this application as described in the "Exemplary Methods" section above. For example, implementing:

[0066] Asynchronous collection of raw log data from multiple sources is performed, and the data is encapsulated into log event messages and delivered to a distributed message queue via a non-blocking interface to obtain the log event messages stored in the queue. Based on the metadata characteristics of the log event messages, the messages are allocated to the corresponding message queues using preset routing rules. An Actor model cluster is used to consume the log event messages in each message queue in parallel to obtain structured log data. During parallel consumption, the number of Actor cluster instances is dynamically adjusted according to thresholds by monitoring the partition backlog of the message queues and the load rate of the Actor model cluster. A supervisor hierarchy is used to monitor the running status of the Actor model cluster, restarting or rerouting faulty Actor models, and handling failed messages using a dead-letter queue.

[0067] The memory 202 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0068] The processor 201 can be an integrated circuit chip with signal processing capabilities. The processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0069] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0070] Exemplary computer-readable storage media and computer program products

[0071] This application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor 201, the computer program implements a message-driven distributed log processing method according to various embodiments of this application as described in the "Exemplary Methods" section above. For example, it implements:

[0072] Asynchronous collection of raw log data from multiple sources is performed, and the data is encapsulated into log event messages and delivered to a distributed message queue via a non-blocking interface to obtain the log event messages stored in the queue. Based on the metadata characteristics of the log event messages, the messages are allocated to the corresponding message queues using preset routing rules. An Actor model cluster is used to consume the log event messages in each message queue in parallel to obtain structured log data. During parallel consumption, the number of Actor cluster instances is dynamically adjusted according to thresholds by monitoring the partition backlog of the message queues and the load rate of the Actor model cluster. A supervisor hierarchy is used to monitor the running status of the Actor model cluster, restarting or rerouting faulty Actor models, and handling failed messages using a dead-letter queue.

[0073] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0074] Furthermore, embodiments of this application can also be computer program products, comprising computer program instructions that, when executed by a processor, implement the steps of a message-driven distributed log processing method according to various embodiments of this application as described in the "Exemplary Methods" section above. For example, implementing:

[0075] Asynchronous collection of raw log data from multiple sources is performed, and the data is encapsulated into log event messages and delivered to a distributed message queue via a non-blocking interface to obtain the log event messages stored in the queue. Based on the metadata characteristics of the log event messages, the messages are allocated to the corresponding message queues using preset routing rules. An Actor model cluster is used to consume the log event messages in each message queue in parallel to obtain structured log data. During parallel consumption, the number of Actor cluster instances is dynamically adjusted according to thresholds by monitoring the partition backlog of the message queues and the load rate of the Actor model cluster. A supervisor hierarchy is used to monitor the running status of the Actor model cluster, restarting or rerouting faulty Actor models, and handling failed messages using a dead-letter queue.

[0076] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0077] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A message-driven based distributed log processing method, characterized in that, The method comprises the following steps: Asynchronously collecting multi-source original log data, encapsulating the original log data into log event messages, and delivering the log event messages to a distributed message queue through a non-blocking interface to obtain queue-stored log event messages; According to the metadata characteristics of the log event messages, the log event messages are distributed to corresponding message queues by using preset routing rules; Parallel consumption processing of the log event messages in each message queue is performed by using an Actor model cluster to obtain structured stored log data; wherein, when performing the parallel consumption processing, the number of instances of the Actor cluster is dynamically adjusted according to a threshold value by monitoring the partition backlog of the message queue and the load rate of the Actor model cluster, and the running state of the Actor model cluster is monitored by a supervisor hierarchical structure, a faulty Actor model is restarted or message rerouting processing is performed, and a dead letter queue is used to process failed messages.

2. The method of claim 1, wherein, The asynchronous collection of the multi-source original log data and the encapsulation of the original log data into log event messages comprise: Filtering, compressing and encrypting the original log data by using a lightweight edge node deployed at a log source end to obtain first log stream data; Performing real-time log model identification and classification on the first log stream data based on a pre-trained neural network model, wherein the neural network model is used to distinguish log types and attach semantic labels; Serializing the log data stream with the attached semantic labels into log event messages in a columnar storage format in a user space, and writing the log event messages into a distributed message queue formed by a remote persistent memory through an RDMA network protocol.

3. The method of claim 1, wherein, The metadata characteristics include log types, source IPs and business priorities.

4. The method according to claim 1 or 3, characterized in that, The distribution of the log event messages to corresponding message queues according to the metadata characteristics of the log event messages by using preset routing rules comprises: Calculating a hash value corresponding to the metadata characteristics of the log event messages according to a consistent hashing algorithm; Mapping the calculated hash value to a message queue partition range to always route log event messages from the same device or user to the same fixed message queue partition.

5. The method of claim 1, wherein, The Actor model cluster comprises a plurality of Actor processing units divided according to functions; the Actor processing units subscribe to corresponding partition queues of the message queue, asynchronously receive log event messages through a fuel tank mechanism, and perform log parsing, compliance verification, business rule execution and storage preprocessing operations.

6. The method according to claim 1 or 5, characterized in that, The parallel consumption processing of the log event messages in each message queue by using the Actor model cluster comprises: parallel consumption processing of the log event messages by the Actor model cluster comprising a parsing unit, a verification unit, a rule engine unit and a storage unit, and sequentially completing format parsing, compliance verification, rule matching and persistent processing.

7. The method of claim 1, wherein, The dynamic adjustment of the number of instances of the Actor cluster according to a threshold value by monitoring the partition backlog of the message queue and the load rate of the Actor model cluster comprises: Real-time monitoring of the partition backlog of each message queue and the load rate of the Actor model cluster; When the partition backlog of any message queue exceeds a first preset threshold, or the load rate of the Actor model cluster exceeds a second preset threshold, a new Actor model is added and assigned to the corresponding message queue; when the partition backlog of any message queue is lower than a third preset threshold, or the load rate of the Actor model cluster is lower than a fourth preset threshold, the corresponding idle Actor model is destroyed; wherein the first preset threshold is greater than the third preset threshold, and the second preset threshold is greater than the fourth preset threshold.

8. The method of claim 1, wherein, The binding dead letter queue processing failure message comprises: The message that still fails to be processed after the maximum number of retries is delivered to a specific dead letter queue; The monitoring service periodically polls the dead letter queue, analyzes, records and triggers an alarm for the messages in the dead letter queue to notify the administrator to manually intervene or configure a new processing rule for retry.

9. A message-driven based distributed log processing system, characterized in that, Comprise: The data source layer is used for asynchronously collecting multi-source original log data, encapsulating the log event messages, and delivering the log event messages to a distributed message queue through a non-blocking interface to obtain the log event messages stored in the queue; The message hub layer is used for distributing the log event messages to corresponding message queues according to the metadata characteristics of the log event messages by using a preset routing rule; The intelligent processing layer is used for performing parallel consumption processing on the log event messages in each message queue by using an Actor model cluster to obtain structured stored log data; wherein, when performing parallel consumption processing, the number of instances of the Actor cluster is dynamically adjusted according to the partition backlog of the message queue and the load rate of the Actor model cluster, and the supervisor hierarchical structure is used to monitor the running state of the Actor model cluster, restart the faulty Actor model, perform message rerouting processing, and process the failure messages in combination with the dead letter queue.

10. The system of claim 9, wherein, The system further comprises: The analysis storage layer is used for scene-based storage and cold data archiving of log data, comprising a real-time storage component, an offline storage component and a cold data archiving component; The application display layer is used for outputting real-time analysis results, comprising a real-time data dashboard, an abnormal alarm component and an intelligent decision support component.

Images