Multi-application log aggregation query system and method

Abstract

This invention discloses a multi-application log aggregation and query system and method, belonging to the field of log management technology. The technical problem it addresses is how to achieve log collection, storage, and querying in application integration scenarios. The system includes a log collector, a queue, a data distributor, a data storage unit, a log queryer, a database, and distributed storage nodes. The log collector listens to log files via fsnotify, reads log data using goroutines and pushes the read log data to the queue, and pushes the log data, along with its corresponding timestamps and metadata, as a log stream to the data distributor via goroutines. The data distributor pushes the log stream to the data storage unit. The data storage unit packages the log data into corresponding storage blocks. The log queryer matches all corresponding storage blocks and selects log data from the storage blocks to return to the client.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of log management, in particular to a multi-application log aggregation query system and method. BACKGROUND

[0002] Under the long-term and continuous investment in informatization construction, a large number of business systems have been built by various government departments. Under the efficient mode of government governance turning to the whole type and service type, the service requirement for the people is getting higher and higher, and the informatization construction urgently needs to be reformed, integrated, reconstructed and improved. The existing informatization systems need to be connected, shared, and integrated to establish an integrated information system. Digital means are introduced to seamlessly integrate internal management and government service, and to manage in an integrated manner to quickly support government service matters, so as to implement a series of reform requirements such as "multi-examination integration" and "multi-certificate integration".

[0003] The construction of government application systems has shown three trends of integration, centralization and publicization. These trends have determined that the system user volume has increased greatly and the scale has become larger and larger. In addition to having the characteristics of high concurrency, high flexibility and high availability, it must also be able to respond quickly, which puts higher requirements on operation and maintenance.

[0004] Log is a key way to locate problems. Under the current trend of government application integration construction, multi-application log aggregation query is an important guarantee measure to ensure the normal operation of business without interruption and to respond to problems in a timely and fast manner. It helps to discover various problems in application operation in a timely manner, reduces the operation and maintenance intensity, and improves the operation and maintenance efficiency.

[0005] In the application integration scenario, how to realize log collection, storage and query is a technical problem to be solved. SUMMARY

[0006] The technical task of the present application is to provide a multi-application log aggregation query system and method to solve the technical problem of how to realize log collection, storage and query in the application integration scenario.

[0007] In a first aspect, the present application provides a multi-application log aggregation query system, comprising a log collector, a queue, a data distributor, a data storage, a log queryor, a database and a distributed storage node.

[0008] The log collector is deployed in the application and is used to listen to log files in a specified directory through fsnotify. For a log file that has changed, the log collector is used to read log data through goroutine and push the read log data to the queue. For log data in the queue, the log collector is used to push the log data, its corresponding timestamp and metadata as a log stream to the data distributor through goroutine.

[0009] The data distributor is used to receive and verify log streams based on timestamps, and push verified log streams to the data storage.

[0010] The data storage is used to match the received log stream with the corresponding storage node and allocate storage blocks based on timestamps and metadata, and to package the log data into the corresponding storage blocks. It is also used to build an index based on the log data and its corresponding timestamps, metadata, storage blocks and storage nodes, and to store the index in the database.

[0011] The log queryer is used to receive query requests sent by the client, and based on the time interval and metadata specified in the query request, to search for indexes and match all corresponding storage blocks from the data storage and distributed storage nodes, and select log data from the storage blocks and return it to the client.

[0012] Preferably, the data storage is used to receive log data in ascending order based on timestamps, and to perform deduplication processing on the log data based on timestamps and log data content;

[0013] For the confirmed accepted log data, the data processing is used to match the log data with the corresponding storage node based on the timestamp and metadata, allocate a storage block on the storage node, compress the log data and package it into the storage block;

[0014] The data processing is used to build an index based on log data and its corresponding timestamps, metadata, storage blocks, and storage nodes, and then store the index in the database.

[0015] Preferably, the data storage device is used to store the verified log stream locally. For log data stored locally, the log data corresponding to the log stream is stored in the corresponding storage block and then deleted from the local storage.

[0016] The data storage device is used to register the timestamps and metadata corresponding to the log data into the hash ring using a set of tokens, and to allocate a storage node for the log data through the hash ring.

[0017] Preferably, the data storage device is used to detect the status of each storage block. If the storage block capacity is full or it has not been updated within a predetermined time, the storage block is marked as read-only. The storage block in read-only state only supports reading log data and does not support writing log data.

[0018] For confirmed log data, the data storage device is used to match the corresponding storage node based on the timestamp and metadata, and to determine whether there is a storage block in the storage node. If there is a storage block in the storage node and the storage block is writable, the log data is compressed and packaged into the storage block. If there is no storage block in the storage node or the existing storage block is read-only, the data storage device is used to create a new storage block in the storage node with a read-write state, and to compress and package the log data into the new storage block. An index is built based on the log data and its corresponding timestamp, metadata, new storage block, and storage node, and the index is stored in the database.

[0019] Preferably, the log queryer is used to receive query requests via a front-end API and push the query requests to a queue;

[0020] The log queryer is used to obtain query requests from the queue through the backend, and based on the time interval and metadata specified in the query request, search for indexes from the data storage and distributed storage nodes and match all corresponding storage blocks, select log data from the storage blocks and push them to the queue, and return the selected log data to the frontend API through the queue.

[0021] Based on the time interval and label specified in the log query request, when retrieving data from the data storage device and distributed storage nodes, the log queryer is used to perform deduplication processing on the selected log data through the backend, and to deduplicate log data returned later with the same timestamp and log data.

[0022] Secondly, the present invention provides a multi-application log aggregation query method, applied to a multi-application log aggregation query system as described in any of the first aspects, the method comprising the following steps:

[0023] The fsnotify function monitors log files in a specified directory. For log files that have changed, goroutines read the log data and push the read log data to a queue.

[0024] For log data in the queue, goroutines push the log data, along with its corresponding timestamps and metadata, as a log stream to the data distributor.

[0025] Based on timestamps, log streams are received and verified through a data distributor, and verified log streams are pushed to the data storage.

[0026] For the verified log stream, based on the timestamp and metadata, the corresponding storage node is matched and storage blocks are allocated through the data storage, and the log data is packaged into the corresponding storage block. An index is built based on the log data and its corresponding timestamp, metadata, storage block and storage node, and the index is stored in the database.

[0027] Based on the time interval and metadata specified in the query request, the log queryer searches for indexes in the data store and distributed storage nodes and matches all corresponding storage blocks, then selects log data from the storage blocks and returns it to the client.

[0028] As a preferred method, log data is received through a data storage device based on the increasing order of timestamps, and the log data is deduplicated based on the timestamps and log data content.

[0029] For the confirmed accepted log data, the corresponding storage node is matched based on the timestamp and metadata, and a storage block is allocated on the storage node. An index is built based on the log data and its corresponding timestamp, metadata, storage block and storage node, and the index is stored in the database.

[0030] The log data is compressed and packaged into storage blocks.

[0031] Preferably, the verified log stream is stored locally. For log data stored locally, the log data corresponding to the log stream is stored in the corresponding storage block and then deleted from the local storage.

[0032] For confirmed log data, the timestamp and metadata corresponding to the log data are registered to the hash ring using a set of tokens, and a storage node is allocated to the log data through the hash ring.

[0033] Preferably, the status of each storage block is detected by the data storage device. If the storage block capacity is full or it has not been updated within a predetermined time, the storage block is marked as read-only. The storage block in read-only state only supports reading log data and does not support writing log data.

[0034] For confirmed log data, based on the timestamp and metadata, the corresponding storage node is matched through the data storage device, and it is determined whether a storage block exists in the storage node. If a storage block exists in the storage node and the storage block is writable, the log data is compressed and packaged into the storage block. If no storage block exists in the storage node or the existing storage block is read-only, a new storage block with a readable and writable state is created in the storage node through the storage processor, and the log data is compressed and packaged into the new storage block. An index is then built based on the log data, its corresponding timestamp, metadata, the new storage block, and the storage node, and the index is stored in the database.

[0035] Preferably, the query request is received from the client through the front-end API of the log queryer and pushed to the queue. Based on the time interval and metadata specified in the query request in the queue, the back-end of the log queryer searches for the index from the data storage and distributed nodes and matches all corresponding storage blocks. Log data is selected from the storage blocks and pushed to the queue. The selected log data is returned to the front-end API of the log queryer through the queue.

[0036] Based on the time interval and label specified in the log query request, when searching for the index and matching the corresponding data block from the data storage and distributed nodes, the backend of the log queryer performs deduplication on the selected log data, and deduplicates the log data returned later that have the same timestamp and log data.

[0037] The multi-application log aggregation query system and method of the present invention have the following advantages:

[0038] 1. The log collector monitors log files. For changed log files, the log data is sent to the data distributor for verification. The verified log data is then stored in multi-node blocks according to timestamps and metadata through the data storage. An index is built for log data, storage blocks, timestamps, metadata, and storage nodes. Based on the time interval and metadata specified in the query request, the corresponding storage block can be matched and the log data selected from the data storage and distributed storage nodes through the index. This realizes the collection, storage, and querying of log data, which can solve problems such as large number of logs, lack of centralization, and difficulty in finding them. It facilitates error location, finding solutions, and helps in service diagnosis and analysis, improves troubleshooting capabilities, enhances application security, stability, and service capabilities, and reduces risks.

[0039] 2. The log data collected by the log collector is pushed to the message queue, which then pushes the log data to the data distributor. At the same time, when performing log queries, the front-end API of the log queryer pushes the query request to the queue, and the returned log data is also pushed to the queue. The queue returns the queried log data to the front-end API of the log queryer. The queue relieves pressure and improves processing efficiency.

[0040] 3. By storing log data in blocks across multiple nodes, the pressure on the database is reduced and the data processing efficiency is improved;

[0041] 4. The log stream is verified by the distributor and sent to the data storage. The data storage receives the verified log stream and stores it locally. Then, it reads the log data from the local storage and performs multi-node block storage, thereby avoiding data loss caused by system crash or power failure.

[0042] 5. When the data storage device receives a verified log stream, it performs deduplication on the log stream to avoid invalid duplicate data storage blocks, saving storage space and improving the accuracy and efficiency of subsequent queries. At the same time, during subsequent queries, the log queryer performs deduplication on the returned log data, improving the accuracy of the query. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] The invention will be further described below with reference to the accompanying drawings.

[0045] Figure 1 This is a flowchart of the log collection process in a multi-application log aggregation and query system, as shown in Example 1.

[0046] Figure 2 This is a flowchart of the log storage and log query process in a multi-application log aggregation and query system, as shown in Example 1.

[0047] Figure 3 Example 2: Flowchart of a multi-application log aggregation query method. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0049] This invention provides a multi-application log aggregation and query system and method to solve the technical problem of how to collect, store and query logs in application integration scenarios.

[0050] Example 1:

[0051] This invention discloses a multi-application log aggregation query system, comprising a log collector, a queue, a data distributor, a data storage device, a log query device, a database, and distributed storage nodes. This system is used to implement multi-application log aggregation queries in application integration scenarios.

[0052] The log collector is deployed in the application. Based on the configured collection rules, the log collector listens to log files via fsnotify. For active log files, it uses goroutines to read them using a tail-f-like process, and sends the read content to a queue. For the log content in the queue, the log collector sends the log data, along with the corresponding timestamps and metadata, as a log stream to the data distributor according to a predetermined period and in batches based on the goroutine's log lines.

[0053] After receiving the pushed log stream, the data distributor parses the log stream and performs correctness verification, including verifying the validity of timestamps and metadata, and pushes the verified log stream to the data storage.

[0054] The data storage receives log data in ascending order of timestamps and performs deduplication on the log data based on the timestamps and log data content. If the incoming line completely matches a previously received line (matches both the timestamp and the log text), the incoming line will be considered a complete duplicate and ignored; if the incoming line has the same timestamp as the previous line but different content, the log line will be accepted.

[0055] After deduplication, for the confirmed log data, the data processor matches the log data with the corresponding storage node based on the timestamp and metadata, allocates a storage block on the storage node, compresses the log data, and packages it into the storage block.

[0056] This embodiment uses consistent hashing to ensure the consistency of the data stream and data processors. They share a hash ring. Data processors register the timestamps and metadata corresponding to log data into the hash ring using a set of tokens. Each token is a random 32-bit unsigned integer. Simultaneously, storage processors report their status to the hash ring, which then allocates a storage node to the log data. Because all storage processors use the same hash ring, write requests can be sent to any node. To ensure consistency, data processors wait to receive responses from at least half plus one storage processor before responding to the client.

[0057] Each storage processor has five lifecycle states: PENDING, JOINING, ACTIVE, LEAVING, and UNHEALTHY. Storage processors in the JOINING and ACTIVE states can accept write requests, while those in the ACTIVE and LEAVING states can accept read requests.

[0058] In this embodiment, the data processor monitors the status of storage blocks in real time. A storage block is compressed and marked as read-only under the following conditions: the block capacity is full, the current block content has not been updated for too long, or the block is refreshed. Whenever a data block is compressed and marked as read-only, a writable data block replaces it. When refreshed to persistent storage, the block is hashed based on its tenant, tag, and content. This means that multiple instances with the same data replica will not write the same data twice to the backup storage; however, if a write to one replica fails, multiple different block objects will be created in the backup storage. Based on this status, the data processor provides storage block detection and log appending services.

[0059] As a specific implementation of the log appending service, for confirmed received log data, the data storage device matches the corresponding storage node based on the timestamp and metadata, and determines whether a storage block exists in the storage node. If a storage block exists in the storage node and the storage block is writable, the log data is compressed and packaged into the storage block. If no storage block exists in the storage node or the existing storage block is read-only, the data storage device creates a new storage block in the storage node with a read-write state, compresses and packages the log data into the new storage block, and builds an index based on the log data, its corresponding timestamp, metadata, the new storage block, and the storage node, and stores the index in the database.

[0060] The data processor temporarily stores data in memory, which may result in data loss in the event of a crash. To address this issue, a data storage device is used to store verified log streams locally. For log data stored locally, the log data corresponding to the log stream is stored in the corresponding storage block and then deleted from local storage. This ensures that verified data is persistently preserved even in the event of a process crash. Upon restart, all data in the logs is replayed, and the device registers itself, ready for subsequent write operations.

[0061] The log queryer receives query requests from clients and, based on the time interval and metadata specified in the query request, searches for indexes in the data store and distributed storage nodes, matches all corresponding storage blocks, and selects log data from the storage blocks to return to the client.

[0062] The log queryer interacts with the client through a front-end API and receives query requests submitted by the client. Based on the time range and tags given in the query request, the log queryer searches the index to determine all matching blocks, then fetches the selected logs and returns the query results.

[0063] As a specific implementation of the query, the log queryer first accesses all data processors to obtain log data stored locally. Only when there is no data matching the conditions in the local storage will it send the same query request to the distributed storage nodes.

[0064] As an improvement to the log query tool, this log query tool has a built-in deduplication function to deduplicat logs with the same timestamp, tag group, and message content.

[0065] As an improvement, the log queryer has a queue between its backend and frontend API. After receiving a query request from the client, the frontend API pushes the query request to the queue. The backend of the log queryer, acting as workers, extracts jobs from the queue and queries the corresponding log data from the data store and distributed storage nodes respectively. The returned log data is pushed to the queue, and the queue returns it to the frontend API. The frontend API then summarizes the data and feeds it back to the client.

[0066] This embodiment provides a multi-application log aggregation and query system that can perform log collection, log storage, and log querying. Specifically:

[0067] (1) Log collection: Deploy the log collector in the application, listen to the log files in the specified directory, read the file content in real time and send it to the queue. The collector will read the log lines in the queue, and push them to the data distributor in batches after adding metadata information such as tags.

[0068] (2) Log storage: After the data distributor verifies the log stream, it pushes the verified log stream to the data storage. The data storage will send the log data to multiple nodes in batches for storage based on the timestamp and metadata. During storage, the log data will be compressed and stored in blocks, and the index built based on the log data, storage blocks, storage nodes, timestamp and metadata will be stored in the database.

[0069] (3) Log query: The system is read-write separated and provides a log queryer. Based on the database index, logs can be queried through metadata and time range. It also has functions such as caching and load balancing to accelerate the query. The system provides a query interface and realizes the aggregated query of logs from multiple applications from the perspective of the application.

[0070] Example 2:

[0071] This invention discloses a multi-application log aggregation query method, applied to the system disclosed in Embodiment 1. The method includes the following steps:

[0072] S100: Monitor log files in a specified directory using fsnotify. For log files that have changed, read the log data using goroutines and push the read log data to a queue.

[0073] For log data in the queue, goroutines push the log data, along with its corresponding timestamps and metadata, as a log stream to the data distributor.

[0074] S200: Based on timestamps, receive and verify log streams through a data distributor, and push verified log streams to the data storage.

[0075] S300: For a verified log stream, based on the timestamp and metadata, match the corresponding storage node through the data storage and allocate storage blocks, package the log data into the corresponding storage block, build an index based on the log data and its corresponding timestamp, metadata, storage block and storage node, and store the index in the database;

[0076] S400: Based on the time interval and metadata specified in the query request, the log queryer searches for indexes in the data store and distributed storage nodes and matches all corresponding storage blocks, then selects log data from the storage blocks and returns it to the client.

[0077] Step S100 implements log collection. In this process, the log collector is deployed in the application. Based on the collection rule configuration, the log collector listens to log files through fsnotify. For active log files, it uses goroutines to read them using a tail-f-like process, and sends the read content to a queue. For the log content in the queue, the log collector sends the log data, along with the corresponding timestamps and metadata, as a log stream to the data distributor according to a predetermined period and in batches based on the goroutine log lines.

[0078] In step S200, after receiving the pushed log stream, the data distributor parses the log stream and performs a correctness check, including verifying the validity of the timestamp and metadata, and pushes the log stream that passes the verification to the data storage.

[0079] Step S300: The data processor implements multi-node distributed storage of log data.

[0080] The specific implementation is as follows:

[0081] (1) Based on the increasing order of timestamps, log data is received through a data storage device, and the log data is deduplicated based on the timestamps and log data content;

[0082] (2) For the confirmed accepted log data, match the corresponding storage node for the log data based on the timestamp and metadata, allocate a storage block on the storage node, build an index based on the log data and its corresponding timestamp, metadata, storage block and storage node, and store the index in the database;

[0083] (3) Compress the log data and package it into a storage block.

[0084] Step (1) performs deduplication on the log data based on the timestamp and log data content. If the incoming line completely matches the previously received line (matches both the previous timestamp and the log text), the incoming line will be considered a complete duplicate and ignored; if the incoming line has the same timestamp as the previous line but different content, the log line will be accepted.

[0085] Step (2) The data processor registers the timestamp and metadata corresponding to the log data into the hash ring through a set of tokens. Each token is a random 32-bit unsigned integer. At the same time, the storage processor reports its status to the hash ring and allocates a storage node for the log data through the hash ring.

[0086] Each storage processor has five lifecycle states: PENDING, JOINING, ACTIVE, LEAVING, and UNHEALTHY. Storage processors in the JOINING and ACTIVE states can accept write requests, while those in the ACTIVE and LEAVING states can accept read requests.

[0087] As an improvement, during step (2), the data processor monitors the status of storage blocks in real time. A storage block is compressed and marked as read-only under the following conditions: the block capacity is full, the current block content has not been updated for too long, or the block is refreshed. Whenever a data block is compressed and marked as read-only, a writable data block replaces it. When refreshed to persistent storage, the block is hashed based on its tenant, tag, and content. This means that multiple instances with the same data replica will not write the same data twice to the backup storage, but if a write to one of the replicas fails, multiple different block objects will be created in the backup storage. Based on this status, the data processor provides storage block detection and log appending services.

[0088] As a specific implementation of the log appending service, for confirmed received log data, the data storage device matches the corresponding storage node based on the timestamp and metadata, and determines whether a storage block exists in the storage node. If a storage block exists in the storage node and the storage block is writable, the log data is compressed and packaged into the storage block. If no storage block exists in the storage node or the existing storage block is read-only, the data storage device creates a new storage block in the storage node with a read-write state, compresses and packages the log data into the new storage block, and builds an index based on the log data, its corresponding timestamp, metadata, the new storage block, and the storage node, and stores the index in the database.

[0089] As an improvement to step S300, the data processor temporarily stores data in memory, which may lead to data loss in the event of a crash. To address this issue, a data storage device is used to store verified log streams locally. For log data stored locally, the log data corresponding to the log stream is stored in the corresponding storage block and then deleted from the local storage. This ensures that verified data is persistently preserved even in the event of a process crash. After restarting, all data in the logs will be replayed, and the device will then register itself, ready for subsequent write operations.

[0090] Step S400 involves performing a log query. Specifically, the log queryer first accesses all data processors through the backend to retrieve locally stored log data. Only when no matching data is found in the locally stored data will the same query request be sent to the distributed storage nodes.

[0091] As an improvement to the log queryer, the backend of the log queryer has a built-in deduplication function to deduplicat logs with the same timestamp, tag group and message content.

[0092] As an improvement, this log queryer interacts with the client via a front-end API to obtain query requests, and then processes the queries through a back-end. A queue exists between the back-end and the front-end API. After receiving a query request from the client, the front-end API pushes the request to the queue. The back-end of the log queryer, acting as workers, retrieves jobs from the queue and queries the corresponding log data from the data store and distributed storage nodes respectively. The returned log data is pushed back to the queue, which then returns it to the front-end API. The front-end API then aggregates the data and sends it back to the client.

[0093] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the means in the different embodiments described above, and these embodiments are also within the protection scope of the present invention.

Claims

1. A multi-application log aggregation query system, characterized in that The log collector, the queue, the data distributor, the data storage, the log queryor, the database and the distributed storage node are included; The log collector is arranged in the application, is used for monitoring the log file in the specified directory through fsnotify, and is used for reading the log data through goroutine and pushing the read log data into the queue for the log data in the queue, and is used for pushing the log data and the corresponding timestamp and metadata as a log stream to the data distributor through goroutine; The data distributor is used for receiving and verifying the log stream based on the timestamp, and pushing the verified log stream to the data storage; The data storage is used for matching the corresponding storage node and allocating the storage block for the received log stream based on the timestamp and the metadata, and is used for packing the log data into the corresponding storage block, and is used for constructing the index based on the log data and the corresponding timestamp, metadata, storage block and storage node, and storing the index into the database; The log queryor is used for receiving the query request sent by the client, and is used for finding the index and matching all the corresponding storage blocks from the data storage and the distributed storage node based on the time interval and the metadata specified in the query request, and is used for selecting the log data from the storage block and returning the client; The data storage is used for receiving the log data based on the increasing order of the timestamp, and is used for carrying out the deduplication processing on the log data based on the timestamp and the log data content; For the confirmed accepted log data, the data processor is used for matching the corresponding storage node for the log data based on the timestamp and the metadata, and is used for allocating a storage block on the storage node, and is used for packing and storing the compressed log data into the storage block; The data processor is used for constructing the index based on the log data and the corresponding timestamp, metadata, storage block and storage node, and storing the index into the database; The data storage is used for storing the verified log stream in the local, and is used for storing the corresponding log data of the log stream into the corresponding storage block and deleting the log data from the local for the log data stored in the local; The data storage is used for registering the timestamp and the metadata of the log data into the hash ring through a group of tokens, and is used for allocating a storage node for the log data through the hash ring; The data storage is used for detecting the state of each storage block, and is used for marking the storage block as a read-only state if the storage block capacity is full or is not updated within a predetermined time, and the read-only storage block only supports reading the log data but does not support writing the log data; The data storage is used for detecting the state of each storage block, and is used for marking the storage block as a read-only state if the storage block capacity is full or is not updated within a predetermined time, and the read-only storage block only supports reading the log data but does not support writing the log data; For the log data confirmed to be received, the data storage is configured to match a corresponding storage node based on a timestamp and metadata, determine whether a storage block exists in the storage node, compress and package the log data into the storage block if the storage block exists in the storage node and a state of the storage block is writable, create a new storage block with a state of being readable and writable in the storage node if the storage block does not exist in the storage node or a state of the storage block is read-only, and compress and package the log data into the new storage block, and construct an index based on the log data, the timestamp corresponding to the log data, the metadata, the new storage block and the storage node, and store the index in a database. The log queryer is configured to receive a query request through a front-end API, and push the query request to a queue. The log queryer is configured to acquire the query request from the queue through a back-end, find an index and match all corresponding storage blocks from the data storage and the distributed storage node based on a time interval and metadata specified in the query request, and select log data from the storage blocks and push the log data to the queue, and return the selected log data to the front-end API through the queue. Based on the time interval and the label specified in the log query request, the log queryer is configured to perform deduplication processing on the selected log data through the back-end, and perform deduplication on log data returned later with the same timestamp and log data.

2. A multi-application log aggregation query method, characterized by The method is applied to the multi-application log aggregation query system of claim 1, and the method comprises the following steps: The fsnotify is used to monitor log files in a specified directory, and for a log file that has changed, a goroutine is used to read log data and push the read log data to a queue. For log data in the queue, a goroutine is used to push the log data and the corresponding timestamp and metadata as a log stream to a data distributor. The data distributor is used to receive and verify the log stream based on the timestamp, and push the verified log stream to a data storage. For the verified log stream, the data storage is used to match a corresponding storage node based on the timestamp and the metadata, allocate a storage block, package the log data into the corresponding storage block, construct an index based on the log data, the timestamp corresponding to the log data, the metadata, the storage block and the storage node, and store the index in a database. Based on the time interval and the metadata specified in the query request, the log queryer is used to find an index and match all corresponding storage blocks from the data storage and the distributed storage node, and select log data from the storage blocks and return the log data to a client. The data storage is used to receive log data based on an increasing order of the timestamp, and perform deduplication processing on the log data based on the timestamp and log data content. For the log data confirmed to be received, the data storage is configured to match a corresponding storage node based on a timestamp and metadata, determine whether a storage block exists in the storage node, compress and package the log data into the storage block if the storage block exists in the storage node and a state of the storage block is writable, create a new storage block with a state of being readable and writable in the storage node if the storage block does not exist in the storage node or a state of the storage block is read-only, and compress and package the log data into the new storage block, and construct an index based on the log data, the timestamp corresponding to the log data, the metadata, the new storage block and the storage node, and store the index in a database. Compress and package the log data into a storage block and store it into the storage; The log stream that passes the verification is stored locally, and for the log data stored locally, the log data corresponding to the log stream is stored into the corresponding storage block and then deleted from the local; For the log data confirmed to be accepted, the timestamp and metadata corresponding to the log data are registered into the hash ring through a set of tokens, and a storage node is assigned to the log data through the hash ring; The state of each storage block is detected through the data storage, and if the storage block capacity is full or has not been updated within a predetermined time, the storage block is marked as read-only, and the read-only storage block only supports reading log data but not writing log data; For the log data confirmed to be accepted, the corresponding storage node is matched based on the timestamp and metadata through the data storage, and it is determined whether there is a storage block in the storage node, if there is a storage block in the storage node and the state of the storage block is writable, the log data is compressed and packaged into the storage block, if there is no storage block in the storage node or the existing storage block is read-only, a new storage block with a state of read-write is created in the storage node through the storage processor, and the log data is compressed and packaged into the new storage block, and an index is constructed based on the log data and the timestamp, metadata, new storage block and storage node corresponding thereto, and the index is stored in the database; The query request is received from the client through the front-end API of the log queryer, and the query request is pushed to the queue, the index is found and all corresponding storage blocks are matched from the data storage and distributed nodes based on the time interval and metadata specified in the query request in the queue through the back-end of the log queryer, and the selected log data is pushed to the queue from the storage block, and the selected log data is returned to the front-end API of the log queryer through the queue; When the index is found and the corresponding data block is matched from the data storage and distributed nodes based on the time interval and label specified in the log query request, the selected log data is processed for deduplication through the back-end of the log queryer, and the log data returned later with the same timestamp and log data is deduplicated.

Images