Log calculation method and system

Abstract

The invention discloses a log calculation method and system. The method comprises the steps that S1, a message queue service generates a corresponding message for a new event; s2, transmitting the message to a system in S3 for object storage; s3, a message queue service notification is called back to a DuckDB server; S4, an event notification is called back, and a corresponding processing flow is triggered; and S5, calculating and processing the log information. After the DuckDB is introduced, a developer only needs to pay attention to the operation state of the single machine, and development and debugging are easy. Deployment links are simplified, the capacity problem does not need to be worried about on the basis of an S3 system, and metadata can be horizontally expanded.

Description

Technical Field

[0001] The present application relates to the field of computer operation and maintenance, and in particular to a log calculation method and system. Background Art

[0002] As cloud computing and AI mature, software systems are becoming increasingly complex. When software system anomalies occur, operations and maintenance personnel need to locate and troubleshoot them. Log computation and analysis are key foundational technologies. Existing log system computation frameworks, typically Spark and Flink, suffer from the following technical drawbacks:

[0003] 1. Depends on Java environment and operating environments such as YARN and Kubernetes, making deployment and operation and maintenance complex.

[0004] 2. The overhead of starting the JVM cluster and task scheduling is high, and the efficiency of processing small-scale data is low.

[0005] 3. State management is highly complex, development and debugging are difficult, and the learning curve is steep.

[0006] In terms of storage, existing big data storage systems are generally based on HDFS. The technical defects of HDFS-based storage systems are as follows:

[0007] 1. Deployment and operation and maintenance are complex, especially when the capacity is large, the maintenance burden is high.

[0008] 2. The number of copies is high, which takes up a lot of resources.

[0009] 3. Metadata is limited by the namenode's limited performance. Summary of the Invention

[0010] In order to solve the technical problems existing in the prior art, this application provides a log calculation method, comprising the following steps:

[0011] S1: The message queue service generates corresponding messages for new events;

[0012] S2: The message is transmitted to the S3 system for object storage;

[0013] S3: The message queue service notifies the DuckDB server:

[0014] S4: Event notification callback triggers the corresponding processing flow;

[0015] S5: Calculate and process the log information.

[0016] Furthermore, in step S2, the message is transmitted to the S3 system for object storage. DuckDB supports batch writing of files, which are stored in the S3 system according to Hive rules for direct reading and filtering of data through DuckDB.

[0017] Furthermore, step S2: the method of writing files in batches includes:

[0018] Pull messages in batches at a scheduled time.

[0019] Organize multiple messages into files.

[0020] Use the Cloud Storage SDK to upload files to the specified S3 system bucket.

[0021] Furthermore, in step S2, the message is transmitted to the S3 system for object storage, and the data from a certain table is directly written to the S3 system and stored as a parquet file.

[0022] Furthermore, step S5: calculating and processing log information also includes consumption calculation: when the parquet file is written to the S3 system, the S3 system service immediately sends a notification message. After the DuckDB service receives the file message, it consumes the single parquet file.

[0023] Furthermore, step S1 specifically includes:

[0024] S11. Set up message queue middleware;

[0025] S12. Define event sources: Determine the events or data changes that need to trigger messages;

[0026] S13. Configure the message publishing mechanism;

[0027] S14. Establish event content.

[0028] Furthermore, S3: the message queue service notification callback to the DuckDB server includes:

[0029] S31: event subscription;

[0030] S32: message push;

[0031] S33: callback message content:

[0032] S34: Confirm the message content.

[0033] The present application also provides a log calculation system for the log calculation method, comprising a message generation unit, a transmission unit, a message callback unit, an event notification unit, and a calculation unit, wherein:

[0034] The message generation unit is used to generate corresponding messages for new events using the message queue service;

[0035] The transmission unit is used to transmit the message to the S3 system for object storage;

[0036] The message callback unit is used to callback the message queue service notification to the DuckDB server:

[0037] The event notification unit is used to call back event notifications and trigger corresponding processing procedures;

[0038] The computing unit is used to calculate and process log information.

[0039] The present application also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the log calculation method is executed.

[0040] The present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the log calculation method as described.

[0041] In actual applications, the modules described in the system and method disclosed in this application can be deployed in multiple modules on one target server, or each module can be independently deployed on a different target server. In particular, as needed, in order to provide more powerful computing processing capabilities, the modules can also be deployed on cluster target servers as needed.

[0042] It can be seen that the technical effects achieved by the technology adopted in this application are:

[0043] 1. After the application introduces DuckDB, developers only need to focus on the running status of a single machine, which simplifies development and debugging.

[0044] 2. Since DuckDB supports SQL, developers can implement business requirements by simply writing SQL.

[0045] 3. DuckDB has high performance, can significantly reduce computing resources and save costs, and only requires a single binary to run without any additional dependencies, simplifying the deployment process.

[0046] 4. Object storage based on the S3 system allows direct use of cloud services without worrying about capacity issues. Object storage can reduce the number of copies through erasure coding technology, and metadata can also be horizontally expanded.

[0047] In order to have a clearer and more comprehensive understanding of the present application, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0049] Figure 1 The following is a flow chart of a log calculation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present application provides an improved technical solution, which is a log calculation method, comprising the following steps:

[0051] S1: The message queue service generates corresponding messages for new events;

[0052] S2: The message is transmitted to the S3 system for object storage;

[0053] S3: The message queue service notifies the DuckDB server:

[0054] S4: Event notification callback triggers the corresponding processing flow;

[0055] S5: Calculate and process the log information.

[0056] See also Figure 1 , Figure 1 In the description, S3 object storage refers to the S3 system that can perform object storage, and DuckDBworker refers to the "working node" or "working thread" responsible for processing messages subscribed to in the message queue service. The technical solution of this application is described in detail in combination with various specific embodiments.

[0057] S1: The message queue service generates corresponding messages for new events.

[0058] The system receives log data from various sources, performs authentication, data cleaning, and data processing, and finally stores it in a unified JSON or CSV format in a message file.

[0059] The message queue service is responsible for sending the subscribed log message notification callback (event notification callback) to the DuckDB server. That is, the message queue notifies the DuckDB server that certain events have occurred through the notification mechanism.

[0060] As an implementation method, the method specifically includes:

[0061] S11. Set up message queue middleware: You can choose middleware such as Kafka, RabbitMQ, or cloud service message queues (such as AWS SQS, Aliyun MQ).

[0062] S12. Define event sources: Identify the events or data changes that need to trigger messages. For example, a log status change, a database write operation, or user behavior.

[0063] S13.Configure the message publishing mechanism:

[0064] Integrate message publishing code at the point where events occur (such as the application or database layer).

[0065] When an event is triggered, the message queue API is called to publish the event information (usually a message in JSON format) to a predefined topic or queue.

[0066] S14. Establish event content:

[0067] Contains necessary information, such as event timestamp, event type, and related data ID or path.

[0068] S2: The message is transmitted to the S3 system.

[0069] DuckDB supports multiple formats, such as JSON, CSV, and Parquet. In terms of efficiency, compression ratio, and performance, the Parquet format is more suitable for this application.

[0070] By using DuckDB's S3 system, data from a table is written directly to the S3 system and stored as parquet files. The following example shows:

[0071] --Install httpfs module

[0072] INSTALL httpfs;

[0073] --Load httpfs module

[0074] LOAD httpfs;

[0075] --Set S3 system key, region and other information

[0076]

[0077] As a preferred embodiment, writing files in batches to the S3 system in step S2 includes:

[0078] Pull messages in batches at regular intervals, such as every few seconds or every minute.

[0079] Organize multiple messages into a file in a choice of formats such as JSON, CSV, or other specialized formats.

[0080] Use a cloud storage SDK (such as AWS SDK, Aliyun OSS SDK, etc.) to upload the file to the specified S3 system bucket.

[0081] As a variation of the implementation method, a direct write method can also be used. The method is to call the storage API when an event is triggered in a real-time application and store each message as a separate object (for example, the naming rule can include a timestamp or a unique ID for sorting and retrieval).

[0082] In the embodiment of the present application, when storing in the S3 system, the data is stored according to the Hive rules, which can be conveniently read and filtered directly through DuckDB. The example is as follows:

[0083]

[0084] S3: The message queue service notifies the DuckDB server.

[0085] The message queue service is responsible for sending subscribed message notification callbacks (event notification callbacks) to the DuckDB server. The message queue uses the notification mechanism to inform the DuckDB server that certain events have occurred, including:

[0086] S31: Event subscription:

[0087] The DuckDB server registers as a subscriber to the message queue and subscribes to a specific topic or queue.

[0088] S32: Message push (push notification):

[0089] When an event occurs, the message queue pushes the message to the subscriber according to the configuration.

[0090] S33: callback message content:

[0091] Implement the message callback function (Consumer) on the server side. After receiving the message, trigger the next step. In the callback function, parse the message content and confirm the message type and related information.

[0092] S34: Confirm message content:

[0093] After successful processing, confirm the message to the message queue to avoid duplication.

[0094] S4: Event notification callback.

[0095] After receiving the notification, the DuckDB server triggers the corresponding processing flow to implement subsequent data retrieval and processing, including:

[0096] Event trigger processing flow: After the message queue pushes the notification, the message handler on the DuckDB server parses the message content.

[0097] Data retrieval: Based on the information in the notification (such as file path, event ID, etc.), obtain the corresponding log data or related information from the S3 system, and use the SDK or API in the S3 system to download the file according to the path or retrieval rules.

[0098] Data preprocessing: Convert, filter, or aggregate stored data to prepare for analysis and calculation.

[0099] Error handling: Set up a fault-tolerance mechanism to ensure automatic retries or logging when access fails.

[0100] S5: Calculate and process the log information.

[0101] Combining the above steps, the system uses the messages, notification mechanisms, and event callbacks stored in the S3 system to process and calculate the log information, including analyzing events, statistical indicators, or generating reports.

[0102] Specific data processing and analysis uses databases (such as DuckDB), data processing libraries (such as pandas), or custom algorithms for analysis, such as through time series statistics, event frequency calculation, anomaly detection, etc.

[0103] In addition, it also includes consumption calculation: when the parquet file is written to the S3 system, the S3 system service immediately sends a notification message. After the DuckDB service receives the file message, it can consume the single parquet file.

[0104] Based on the above embodiment, a log calculation system according to an embodiment of the present application is used to implement the log calculation method, including a message generation unit, a transmission unit, a message callback unit, an event notification unit, and a calculation unit, wherein:

[0105] The message generation unit is used to generate corresponding messages for new events using the message queue service;

[0106] The transmission unit is used to transmit the message to the S3 system for object storage;

[0107] The message callback unit is used to callback the message queue service notification to the DuckDB server:

[0108] The event notification unit is used to call back event notifications and trigger corresponding processing procedures;

[0109] The computing unit is used to calculate and process log information.

[0110] An embodiment of the present application further provides a storage medium, wherein a computer program is stored on the storage medium, and when the computer program is run by a processor, the log calculation method is executed.

[0111] An embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the log calculation method as described.

[0112] It should be noted that, those skilled in the art can understand that all or part of the split steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0113] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log calculation method, characterized in that it includes the steps of: S1: The message queue service generates corresponding messages for new events; S2: The message is transmitted to the S3 system for object storage; S3: The message queue service notifies the DuckDB server: S4: Event notification callback triggers the corresponding processing flow; S5: Calculate and process the log information.

2. The log calculation method according to claim 1, wherein: Step S2: The message is transferred to the S3 system for object storage. DuckDB supports batch writing of files. The data stored in the S3 system is stored according to Hive rules and is used to directly read and filter data through DuckDB.

3. The log calculation method according to claim 1, wherein: Step S2: The method of writing files in batches includes: Pull messages in batches at regular intervals; Organize multiple messages into files; Use the Cloud Storage SDK to upload files to the specified S3 system bucket.

4. The log calculation method according to claim 1, wherein: Step S2: The message is transferred to the S3 system for object storage. The data from a table is written directly to the S3 system and stored as a parquet file.

5. The log calculation method according to claim 3, wherein: Step S5: The calculation and processing of log information also includes consumption calculation: when the parquet file is written to the S3 system, the S3 system service immediately sends a notification message. After the DuckDB service receives the file message, it consumes the single parquet file.

6. The log calculation method according to claim 3, wherein: Step S1 specifically includes: S11. Set up message queue middleware; S12. Define event sources: Determine the events or data changes that need to trigger messages; S13. Configure the message publishing mechanism; S14. Establish event content.

7. The log calculation method according to claim 1, wherein S3 : The message queue service notification callback to the DuckDB server includes: S31: event subscription; S32: message push; S33: callback message content: S34: Confirm the message content.

8. A log calculation system for implementing the log calculation method according to any one of claims 1 to 7, characterized in that: It includes a message generation unit, a transmission unit, a message callback unit, an event notification unit, and a calculation unit, wherein: The message generation unit is used to generate corresponding messages for new events using the message queue service; The transmission unit is used to transmit the message to the S3 system for object storage; The message callback unit is used to callback the message queue service notification to the DuckDB server: The event notification unit is used to call back event notifications and trigger corresponding processing procedures; The computing unit is used to calculate and process log information.

9. A storage medium having a computer program stored thereon, wherein the computer program is operable to execute the log calculation method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device, characterized in that: include: A processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the log calculation method according to any one of claims 1 to 7.

Images