Log collection remote synchronization and persistence method and system based on KAIWU database
The log collection and synchronization method for the KAIWU database addresses the challenge of log persistence and synchronization by using Read and Write Handlers to efficiently manage and distribute log information across multiple sources and destinations, ensuring continuous and unified log processing and display.
Patent Information
- Application Number
- CN202310099082.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-09
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2043-02-09
AI Technical Summary
The KAIWU database faces challenges in continuously obtaining log information from shared memory and remote synchronization, persisting logs to local files, and synchronizing them to remote and external systems, particularly due to the absence of hard disks on end nodes and the need for efficient log aggregation across multiple processes.
A log collection and remote synchronization method using Read Handlers and Server Thread to obtain log information from shared memory and remote sources, persisting it to local files, and synchronizing it to various destinations, implemented through Read Handlers Init(), Read(), and Destory() methods for reading, and Write Handlers Init(), write(), and Destory() methods for writing.
The solution enables continuous log information acquisition, central processing, and synchronized persistence to local files and remote systems, facilitating unified log display and easy expansion by adding new sources and destinations.
Smart Images
Figure CN116170298B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of operation and maintenance monitoring, and in particular to a log collection remote synchronization and persistence method and system based on a KAIWU database. Background Art
[0002] The KAIWU database follows the design concept of Share-Nothing and adopts a fully peer-to-peer architecture, allowing each node to participate in data storage and computing; it supports online horizontal expansion of nodes, while ensuring business continuity, achieving a linear improvement in product performance, and can support the storage and management of PB-level data in the big data era. At the same time, the database has the characteristics of cloud native, multi-center, high availability, strong transaction consistency, etc., supports mixed row and column storage, and meets the needs of HTAP scenarios.
[0003] The current business scope has covered multiple industries such as industry, government affairs, energy, and finance: distributed databases have been implemented in multiple projects such as government affairs and finance; the Kaiwu data service platform based on distributed databases focuses on rapidly developing fields such as industrial Internet and digital energy, and is supported by an integrated cloud-edge-end architecture to vigorously promote the digital transformation of industrial enterprises and the construction of digital energy.
[0004] The KAIWU database is divided into multiple node processes on the cloud edge. The node process on the end does not have a hard disk and only generates logs but cannot persist the logs. The logs need to be synchronized to the edge for persistence. At the same time, the edge can also synchronize logs to the cloud. The end logs are synchronized to the edge, and the edge logs are synchronized to the cloud by sending them over the network.
[0005] At the same time, different processes on the end need to aggregate logs to the main process, and different processes on the edge also need to aggregate logs to the main process. The log transmission method between different processes on the edge and the end adopts shared memory.
[0006] Therefore, the internal logs of the KAIWU database have the characteristics of multiple sources and multiple destinations. Therefore, how to continuously obtain log information from shared memory and remote synchronization within the KAIWU database, persist the log information to local files, output it to the console, and continue to synchronize it to remote and external systems is a technical problem that needs to be solved urgently. Summary of the invention
[0007] The technical task of the present invention is to provide a log collection, remote synchronization and persistence method and system based on the KAIWU database to solve the problem of how to continuously obtain log information from shared memory and remote synchronization within the KAIWU database, and persist the log information to local files, output it to the console, and continue to synchronize it to remote and external systems.
[0008] The technical task of the present invention is achieved in the following way: a log collection remote synchronization and persistence method based on KAIWU database, which uses Read Handlers, Server Thread and Write Handlers to continuously obtain log information from shared memory and remote synchronization inside the KAIWU database, and persist the log information to local files, output to the console and continue to synchronize to the remote and external systems; the details are as follows:
[0009] Read log information from various sources through Read Handlers;
[0010] The server thread traverses each read handler to obtain log information and sends it to each write handler;
[0011] Receive log information through Write Handlers and write log information to different destinations.
[0012] Preferably, Read Handlers is a collection of all source logs, and each implementation of reading log sources is called a Read Handler;
[0013] Read Handlers include at least one Read handler for reading logs from shared memory and one Read handler for reading logs sent remotely.
[0014] Preferably, the Read Handlers are implemented based on the Read Handlers interface, which includes a Read Handlers Init() method, a Read Handlers Read() method and a Read Handlers Destory() method;
[0015] Among them, the Read Handlers Init() method is used to initialize the Read Handler;
[0016] Read Handlers Read() method is used for Server Thread to call and return log data in a fixed format to Server Thread;
[0017] The Read Handlers Destory() method is used to destroy the Read Handler object.
[0018] Preferably, Write Handlers is a set of implementations that send logs to all destinations. The implementation that sends logs to each destination is called a Write Handler.
[0019] Write Handlers include at least one Write Handler that sends logs to shared memory and one Write Handler that sends logs to a remote service.
[0020] Preferably, Write Handlers are implemented based on the same Write Handlers interface, and the Write Handlers interface includes a Write Handlers Init() method, a Write Handlers write() method, and a Write Handlers Destory() method;
[0021] The Write Handlers Init() method is used to initialize the Write Handler;
[0022] Write Handlers write() method is used to call Server Thread, receive the fixed format log written by Server Thread, and write the log data to the corresponding destination;
[0023] The Write Handlers Destory() method is used to destroy the Write Handler object.
[0024] As a preferred embodiment, the Server Thread is the manager of the Read Handlers and Write Handlers. When the service is started, the Server Thread is used to load the specified Read Handlers and Write Handlers, and then periodically traverse each Read Handlers, and then send the read log information to each Write Handlers respectively;
[0025] Preferably, the Server Thread is a separate thread that performs periodic tasks; the Server Thread includes a Read Handlers list and a Write Handlers list;
[0026] The Read Handlers list includes all Read Handlers registered when the system starts;
[0027] The Write Handlers list includes all the Write Handlers registered when the system starts;
[0028] The execution process of each cycle of the Server Thread thread is as follows:
[0029] Call the Read Handlers read() method of each Read Handler registered in the Read Handlers list to read the log data;
[0030] Call the Write Handlerswrite() method of each Write Handler registered in the Write Handlers list respectively, and send the read logs to the target location corresponding to the Write handler.
[0031] A log collection remote synchronization and persistence system based on KAIWU database, the system includes a ReadHandlers unit for reading log information from various sources;
[0032] The Server Thread unit is used to traverse each Read Handler to obtain log information and send it to each Write Handler;
[0033] The Write Handlers unit is used to receive log information and write it to different destinations.
[0034] An electronic device comprising: a memory and at least one processor;
[0035] Wherein, the memory stores a computer program;
[0036] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the log collection remote synchronization and persistence method based on the KAIWU database as described above.
[0037] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the log collection remote synchronization and persistence method based on the KAIWU database as described above.
[0038] The log collection remote synchronization and persistence method and system based on KAIWU database of the present invention have the following advantages:
[0039] (1) The present invention solves the problem of remote synchronization and persistence of KAIWU database log collection, and centrally collects and persists the log information generated by each process of the KAIWU database cluster;
[0040] (ii) The present invention can realize the continuous acquisition of log information from shared memory and remote synchronization within the KAIWU database, and persist the log information to local files, output to the console, and continue to synchronize to remote and external systems;
[0041] (III) The present invention can achieve the effect of unified processing by centrally processing logs from different sources and sending them to different output terminals for display and persistence. At the same time, the present invention is conducive to the expansion of the solution. When adding new sources and destinations, only the corresponding read handler and write handler need to be added. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The present invention is further described below in conjunction with the accompanying drawings.
[0043] Attached Figure 1 This is a schematic diagram of the log collection remote synchronization and persistence method based on the KAIWU database;
[0044] Attached Figure 2 This is a schematic diagram of Read Handlers;
[0045] Attached Figure 3 This is a schematic diagram of Write Handlers;
[0046] Attached Figure 4 This is a schematic diagram of Server Thread. DETAILED DESCRIPTION
[0047] The log collection remote synchronization and persistence method and system based on KAIWU database of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.
[0048] Embodiment 1:
[0049] As attached Figure 1 As shown, this embodiment provides a log collection remote synchronization and persistence method based on the KAIWU database. The method uses Read Handlers, Server Thread and Write Handlers to continuously obtain log information from shared memory and remote synchronization inside the KAIWU database, and persist the log information to a local file, output it to the console, and continue to synchronize with the remote and external systems; the details are as follows:
[0050] S1. Read log information from various sources through Read Handlers;
[0051] S2, traverse each Read Handler through the Server Thread to obtain log information and send it to each Write Handler;
[0052] S3. Receive log information through Write Handlers and write the log information to different destinations.
[0053] As attached Figure 2 As shown, the Read Handlers in this embodiment are a collection of reading all source logs, and each implementation of reading log sources is called a Read Handler;
[0054] Read Handlers include at least one Read handler for reading logs from shared memory and one Read handler for reading logs sent remotely.
[0055] The Read Handlers in this embodiment are implemented based on the Read Handlers interface, which includes the Read Handlers Init() method, the Read Handlers Read() method and the Read Handlers Destory() method;
[0056] Among them, the Read Handlers Init() method is used to initialize the Read Handler;
[0057] Read Handlers Read() method is used for Server Thread to call and return log data in a fixed format to Server Thread;
[0058] The Read Handlers Destory() method is used to destroy the Read Handler object.
[0059] As attached Figure 3 As shown, the Write Handlers in this embodiment are a set of implementations for sending logs to all destinations, and the implementation for sending logs to each destination is called a Write Handler;
[0060] Write Handlers include at least one Write Handler that sends logs to shared memory and one Write Handler that sends logs to a remote service.
[0061] The Write Handlers in this embodiment are implemented based on the same Write Handlers interface. The Write Handlers interface includes the Write Handlers Init() method, the Write Handlers write() method, and the Write Handlers Destory() method.
[0062] The Write Handlers Init() method is used to initialize the Write Handler;
[0063] Write Handlers write() method is used to call Server Thread, receive the fixed format log written by Server Thread, and write the log data to the corresponding destination;
[0064] The Write Handlers Destory() method is used to destroy the Write Handler object.
[0065] As attached Figure 4 As shown, the Server Thread in this embodiment is the manager of the Read Handlers and Write Handlers. When the service is started, the Server Thread is used to load the specified Read Handlers and Write Handlers, and then periodically traverse each Read Handlers, and then send the read log information to each Write Handlers respectively;
[0066] The Server Thread in this embodiment is a separate thread that executes periodic tasks; the Server Thread includes a Read Handlers list and a Write Handlers list;
[0067] The Read Handlers list includes all Read Handlers registered when the system starts;
[0068] The Write Handlers list includes all the Write Handlers registered when the system starts;
[0069] The execution process of each cycle of the Server Thread thread in this embodiment is as follows:
[0070] (1) Call the Read Handlersread() method of each Read Handler registered in the Read Handlers list to read the log data;
[0071] (2) Call the WriteHandlers write() method of each Write Handler registered in the Write Handlers list respectively, and send the read logs to the target location corresponding to the Write handler.
[0072] Embodiment 2:
[0073] This embodiment provides a log collection remote synchronization and persistence system based on the KAIWU database, the system comprising:
[0074] Read Handlers unit, used to read log information from various sources;
[0075] The Server Thread unit is used to traverse each Read Handler to obtain log information and send it to each Write Handler;
[0076] The Write Handlers unit is used to receive log information and write it to different destinations.
[0077] Embodiment 3:
[0078] An embodiment of the present invention further provides an electronic device, comprising: a memory and a processor;
[0079] Wherein, the memory stores computer-executable instructions;
[0080] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the log collection remote synchronization and persistence method based on the KAIWU database in any embodiment of the present invention.
[0081] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.
[0082] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.
[0083] Embodiment 4:
[0084] The embodiment of the present invention also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor, so that the processor executes the log collection remote synchronization and persistence method based on the KAIWU database in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which the software program code that implements the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.
[0085] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0086] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.
[0087] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0088] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A log collection remote synchronization and persistence method based on KAIWU database, characterized in that: This method uses Read Handlers, Server Thread and Write Handlers to continuously obtain log information from shared memory and remote synchronization within the KAIWU database, persist the log information to local files, output it to the console, and continue to synchronize it to remote and external systems; the details are as follows: Read log information from various sources through Read Handlers; The server thread traverses each read handler to obtain log information and sends it to each write handler; Receive log information through Write Handlers and write log information to different destinations; Among them, Server Thread is a separate thread that performs periodic tasks; Server Thread includes ReadHandlers list and Write Handlers list; The Read Handlers list includes all Read Handlers registered when the system starts; The Write Handlers list includes all the Write Handlers registered when the system starts; The execution process of each cycle of the Server Thread thread is as follows: Call the Read Handlers read() method of each Read Handler registered in the Read Handlers list to read the log data; Call the Write Handlerswrite() method of each Write Handler registered in the Write Handlers list respectively, and send the read logs to the target location corresponding to the Write handler.
2. According to the log collection remote synchronization and persistence method based on KAIWU database according to claim 1, it is characterized in that: Read Handlers is a collection of all source logs. Each implementation of reading log sources is called a Read Handler. Read Handlers include at least one Read handler for reading logs from shared memory and one Read handler for reading logs sent remotely.
3. The log collection remote synchronization and persistence method based on the KAIWU database according to claim 1 or 2 is characterized in that: Read Handlers is implemented based on the Read Handlers interface, which includes the ReadHandlers Init() method, the Read Handlers Read() method, and the Read Handlers Destory() method; Among them, the Read Handlers Init() method is used to initialize the Read Handler; Read Handlers Read() method is used by Server Thread to call and return log data in a fixed format to Server Thread; The Read Handlers Destory () method is used to destroy the Read Handler object.
4. According to claim 1, the log collection remote synchronization and persistence method based on the KAIWU database is characterized in that: Write Handlers is a collection of implementations that send logs to all destinations. The implementation that sends logs to each destination is called a Write Handler. Write Handlers include at least one Write Handler that sends logs to shared memory and one Write Handler that sends logs to a remote service.
5. The log collection remote synchronization and persistence method based on KAIWU database according to claim 1 or 4 is characterized in that: Write Handlers are implemented based on the same Write Handlers interface, which includes the Write Handlers Init() method, the Write Handlers write() method, and the Write Handlers Destory() method. The Write Handlers Init() method is used to initialize the Write Handler; Write Handlers write() method is used to call Server Thread, receive the fixed format log written by Server Thread, and write the log data to the corresponding destination; Write Handlers Destory() method is used to destroy the Write Handler object.
6. According to claim 1, the log collection remote synchronization and persistence method based on KAIWU database is characterized in that: Server Thread is the manager of Read Handlers and Write Handlers. When the service starts, Server Thread is used to load the specified Read Handlers and Write Handlers, and then periodically traverse each Read Handlers, and then send the read log information to each Write Handler.
7. A log collection remote synchronization and persistence system based on KAIWU database, characterized in that: The system is used to implement the log collection remote synchronization and persistence method based on the KAIWU database as described in any one of claims 1 to 6; the system includes: Read Handlers unit, used to read log information from various sources; The Server Thread unit is used to traverse each Read Handler to obtain log information and send it to each Write Handler; The Write Handlers unit is used to receive log information and write the log information to different destinations.
8. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the log collection remote synchronization and persistence method based on the KAIWU database as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the log collection remote synchronization and persistence method based on the KAIWU database as described in any one of claims 1 to 6.