Log storage and retrieval method, system, terminal device, and medium

Abstract

The application discloses a log storage and retrieval method, system, terminal equipment and computer readable storage medium. The steps of the log storage and retrieval method include: receiving log data and writing the log data into a preset disk space in parallel compression; receiving a search text sent by a user and determining a target search text corresponding to the search text; determining a to-be-searched file list from the disk space according to a preset search time range, decompressing and reading target log data matched with the target search text from the to-be-searched file list, and storing the target log data into a preset result set. The application can realize efficient storage and retrieval of ten-million-level throughput logs on the basis of saving storage and retrieval costs.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, terminal device, and computer-readable storage medium for storing and retrieving logs. Background Technology

[0002] When centrally storing and retrieving logs output after application containerization, technologies such as ELK*1 (a web-based log analysis tool) and HDFS MapReduce (a software framework upon which applications can be easily written to run on large clusters of thousands of commercial machines, processing terabyte-scale datasets in parallel in a reliable and fault-tolerant manner) are commonly used to centrally store logs and achieve near real-time log retrieval. However, when the log volume per second reaches the GB level, the above solutions require a significant investment of resources to meet storage and retrieval demands. For example, using ELK technology to store and retrieve 10GB of logs per second (approximately 20 million logs per second) would require approximately (2.3GHz * 80,000 CPU cores, 900TB of storage space) * 2.

[0003] Since container application log storage and querying is a write-heavy, read-light scenario, performing word segmentation and inverted indexing on each log entry consumes a lot of CPU and memory resources. However, the overall index utilization rate is not high, and the query volume is only one in a billion of the write volume. It is obviously unreasonable to invest a lot of resources for a relatively small number of search needs, resulting in low overall write performance, high performance tuning difficulty, and difficulty in meeting the requirements of near real-time writing and querying. Summary of the Invention

[0004] The main objective of this invention is to provide a method, system, terminal device, and computer-readable storage medium for storing and retrieving logs, aiming to achieve efficient storage and retrieval of logs with a throughput of tens of millions while saving storage and retrieval costs.

[0005] To achieve the above objectives, the present invention provides a log storage and retrieval method, the log storage and retrieval method comprising:

[0006] Receive log data and temporarily store the log data in a memory cache;

[0007] Receive log data and compress the log data in parallel and write it to a preset disk space;

[0008] Receive the search text sent by the user and determine the target search text corresponding to the search text; determine the list of files to be searched from the disk space according to the preset search time range, and read the target log data that matches the target search text from the list of files to be searched, and store the target log data into the preset result set.

[0009] Optionally, the step of compressing and writing the log data to a preset disk space in parallel includes:

[0010] The log data is temporarily stored in a memory cache, and the log data in the memory cache is compressed and written in parallel to a preset disk space.

[0011] Optionally, the step of temporarily storing the log data in a memory cache and compressing the log data in the memory cache in parallel and writing it to a preset disk space includes:

[0012] The log parsing module receives log data, parses the log data to obtain log data in string format, and sends the log data in string format to the log compression and storage module.

[0013] The log compression and storage module extracts the log type, application name, and log timestamp from the string-format log data, and categorizes and temporarily stores the log data in the memory cache according to the log type, application name, and log timestamp.

[0014] When the number of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed at a preset refresh frequency, the log data in the memory cache is pushed to a preset compression queue through the log compression storage module.

[0015] Log data in the preset compression queue is compressed and written to the disk space in parallel according to the preset log writing rules.

[0016] Optionally, before the steps of receiving log data and compressing and writing the log data in parallel to a preset disk space, the method further includes:

[0017] The gateway module establishes a connection between the log parsing module and the log compression and storage module.

[0018] Optionally, the step of receiving the search text sent by the user and determining the target search text corresponding to the search text includes:

[0019] The gateway module receives the search text sent by the user and performs text syntax verification and text operator recognition on the search text.

[0020] The target search text is obtained by constructing a search tree and logical matching strings based on the search text after text syntax verification and text operator recognition.

[0021] Optionally, the step of determining the list of files to be searched from the disk space according to a preset search time range includes:

[0022] The time range of the logs is determined based on the file names of the logs in the disk space. The target time range is obtained by intersecting the time range with the preset search time range through the query module.

[0023] The list of files to be searched is obtained by retrieving log data corresponding to the target time range from the disk space.

[0024] Optionally, the step of reading target log data matching the target search text from the list of files to be searched includes:

[0025] Compare the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file;

[0026] If the currently read character matches any character in the character group, the currently read character is temporarily stored in a preset log cache, and the preset pointer is pointed to the next character in the character group that matches the currently read character.

[0027] The target word in the log cache is determined, and when all characters in the target word are matched correctly, a logical operation is performed on the logical matching string according to the logical value of the target word in the corresponding logical matching string. When the result of the logical operation is true, the target log data is obtained, and the log cache is cleared.

[0028] After the step of comparing the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file, the method further includes:

[0029] If the currently read character fails to match any of the characters in the character group, the preset pointer is reset.

[0030] Optionally, after the step of storing the target log data into a preset result set, the method further includes:

[0031] The gateway module merges and summarizes the preset result set in chronological order, returns the merged and summarized search results to the Nginx server mounted on the gateway module, and ends the search task.

[0032] To achieve the above objectives, the present invention also provides a log storage and retrieval system, the log storage and retrieval system comprising:

[0033] A compression writing module is used to receive log data and compress the log data in parallel and write it to a preset disk space;

[0034] The receiving module is used to receive the search text sent by the user and determine the target search text corresponding to the search text;

[0035] The retrieval module is used to determine a list of files to be searched from the disk space according to a preset search time range, decompress and read target log data that matches the target search text from the list of files to be searched, and store the target log data into a preset result set.

[0036] In this invention, each functional module of the log storage and retrieval system implements the steps of the log storage and retrieval method described above during operation.

[0037] To achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a log storage and retrieval program stored in the memory and executable on the processor, wherein the log storage and retrieval program, when executed by the processor, implements the steps of the log storage and retrieval method as described above.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a log storage and retrieval program, wherein the log storage and retrieval program, when executed by a processor, implements the steps of the log storage and retrieval method described above.

[0039] In addition, to achieve the above objectives, the present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the log storage and retrieval method as described above.

[0040] This invention provides a log storage and retrieval method, system, terminal device, computer-readable storage medium, and computer program product. The log storage and retrieval method includes the following steps: receiving log data and compressing the log data in parallel and writing it to a preset disk space; receiving search text sent by a user and determining the target search text corresponding to the search text; determining a list of files to be searched from the disk space according to a preset search time range, decompressing and reading target log data matching the target search text from the list of files to be searched, and storing the target log data into a preset result set.

[0041] Compared to existing storage and retrieval methods, this invention compresses and writes received log data to a pre-defined disk space, identifies log data from this disk space that matches the search text sent by the user, and sends this log data to the corresponding search request initiator. Therefore, this invention achieves near real-time parallel compression and writing, saving significant storage space and thus drastically reducing disk I / O operations. Furthermore, this invention utilizes a time-based indexing method to achieve near real-time decompression retrieval and millisecond-level response, greatly improving search speed, similarly significantly reducing disk I / O operations, and meeting the vast majority of search requirements.

[0042] Based on this, the present invention does not require special hardware configuration and SSD solid-state storage disks, and can run on ordinary servers (HDD storage) or virtual machines, which can effectively reduce hardware procurement costs. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiments of the present invention;

[0044] Figure 2 This is a schematic diagram of the first process of an embodiment of the log storage and retrieval method of the present invention;

[0045] Figure 3 This is a second flowchart illustrating an embodiment of the log storage and retrieval method of the present invention;

[0046] Figure 4 This is a schematic diagram of the storage and retrieval structure of an embodiment of the log storage and retrieval method of the present invention;

[0047] Figure 5 This is a schematic diagram of the Receiver module structure according to an embodiment of the log storage and retrieval method of the present invention;

[0048] Figure 6 This is a schematic diagram of the search tree construction in an embodiment of the log storage and retrieval method of the present invention;

[0049] Figure 7 This is a schematic diagram of the Gateway & Searcher module structure according to an embodiment of the log storage and retrieval method of the present invention.

[0050] Figure 8 This is a schematic diagram of the functional modules of an embodiment of the log storage and retrieval system of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0053] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0054] It should be noted that the terminal device in the embodiments of the present invention can be a terminal device used to achieve log storage and retrieval at a throughput of tens of millions. Specifically, the terminal device can be a mobile phone, computer, server, or network device, etc.

[0055] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1 As shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a log storage and retrieval program. The operating system is a program that manages and controls the device's hardware and software resources, supporting the log storage and retrieval program and the operation of other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the log storage and retrieval program stored in the memory 1005 and perform the following operations:

[0058] Receive log data and compress the log data in parallel and write it to a preset disk space;

[0059] Receive the search text sent by the user and determine the target search text corresponding to the search text;

[0060] A list of files to be searched is determined from the disk space according to a preset search time range, and target log data matching the target search text is read from the list of files to be searched and stored in a preset result set.

[0061] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0062] The log data is temporarily stored in a memory cache, and the log data in the memory cache is compressed and written in parallel to a preset disk space.

[0063] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0064] The log parsing module receives log data, parses the log data to obtain log data in string format, and sends the log data in string format to the log compression and storage module.

[0065] The log compression and storage module extracts the log type, application name, and log timestamp from the string-format log data, and categorizes and temporarily stores the log data in the memory cache according to the log type, application name, and log timestamp.

[0066] When the number of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed at a preset refresh frequency, the log data in the memory cache is pushed to a preset compression queue through the log compression storage module.

[0067] Log data in the preset compression queue is compressed and written to the disk space in parallel according to the preset log writing rules.

[0068] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0069] The gateway module establishes a connection between the log parsing module and the log compression and storage module.

[0070] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0071] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0072] The gateway module receives the search text sent by the user and performs text syntax verification and text operator recognition on the search text.

[0073] The target search text is obtained by constructing a search tree and logical matching strings based on the search text after text syntax verification and text operator recognition.

[0074] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0075] The time range of the logs is determined based on the file names of the logs in the disk space. The target time range is obtained by intersecting the time range with the preset search time range through the query module.

[0076] The list of files to be searched is obtained by retrieving log data corresponding to the target time range from the disk space.

[0077] Furthermore, the processor 1001 can also be used to call the log storage retrieval program stored in the memory 1005, and also perform the following operations:

[0078] The query module performs multi-threaded parallel decompression on the list of files to be searched to obtain a data character stream, and stores the data character stream into the corresponding cache.

[0079] Identify data character streams in the cache that are within the preset search time range, and read the data character streams sequentially.

[0080] Compare the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file;

[0081] If the currently read character matches any character in the character group, the currently read character is temporarily stored in a preset log cache, and the preset pointer is pointed to the next character in the character group that matches the currently read character.

[0082] The target word in the log cache is determined, and when all characters in the target word are matched correctly, a logical operation is performed on the logical matching string according to the logical value of the target word in the corresponding logical matching string. When the result of the logical operation is true, the target log data is obtained, and the log cache is cleared.

[0083] After comparing the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file, the processor 1001 can also call the log storage retrieval program stored in the memory 1005, and perform the following operations:

[0084] If the currently read character fails to match any of the characters in the character group, the preset pointer is reset.

[0085] Furthermore, after the step of storing the target log data into a preset result set, the processor 1001 can also call the log storage retrieval program stored in the memory 1005, and perform the following operations:

[0086] The gateway module merges and summarizes the preset result set in chronological order, returns the merged and summarized search results to the Nginx server mounted on the gateway module, and ends the search task.

[0087] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the log storage and retrieval method of the present invention.

[0088] This embodiment provides an example of a log storage and retrieval method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0089] This embodiment employs a low-cost method for storing and retrieving massive logs, which can solve the problems of high resource consumption and low query performance encountered when storing and retrieving logs with tens of millions of TPS. The limitations of existing storage indexing technologies are illustrated using Elastic Search index writing methods and HDFS storage writing methods as examples.

[0090] Regarding Elastic Search index writing:

[0091] To quickly locate target documents, Elastic Search (ES) uses inverted indexes to optimize search speed. Although this consumes more space, the performance improvement is significant. However, due to the massive number of indexes, ES cannot directly fit the entire index into memory. Instead, it creates a dictionary index and constructs a Finite State Transformer (FST) to fit into memory, further improving search efficiency. The space consumption of document IDs within the dictionary is also substantial. ES uses Frame of Reference (FORM) technology to compress the posting list, resulting in a significant compression effect. Furthermore, ES's filter statements employ Roaring Bitmap technology to cache search results, ensuring fast high-frequency filter queries while reducing storage space consumption.

[0092] Regarding HDFS storage writes:

[0093] Hadoop implements a distributed file system (Hadoop Distributed File System, HDFS). HDFS is highly fault-tolerant and designed to be deployed on low-cost hardware, providing high throughput access to application data, making it suitable for applications with very large datasets. HDFS relaxes POSIX requirements, allowing streaming access to data within the file system. The core design of the Hadoop framework is HDFS and MapReduce. HDFS provides storage for massive amounts of data, while MapReduce provides computation for massive amounts of data. Although Hadoop, as a fundamental big data processing platform, has gained widespread recognition for its application value, it still faces many challenges. The main ones are as follows:

[0094] (1) Namenode / jobtracker single point of failure.

[0095] Hadoop uses a master / slaves architecture, which is relatively simple to manage, but has fatal single points of failure and insufficient space capacity, which has seriously affected Hadoop's scalability.

[0096] (2) HDFS small file problem.

[0097] In HDFS, every block, file, or directory is stored in memory as an object, with each object occupying approximately 150 bytes. If there are 10,000,000 small files, each occupying one block, then the namenode (the file system's management node) would require 2GB of space. If storing 100 million files, the namenode would require 20GB of space. Thus, the namenode's memory capacity severely limits the cluster's scalability.

[0098] (3) The jobtracker (the centralized processing point for cluster transactions) performs both monitoring and scheduling, resulting in excessive load. To address this issue, Yahoo has begun designing the next generation of Hadoop MapReduce. The main idea is to separate monitoring and scheduling, creating a dedicated monitoring component, while the jobtracker is only responsible for overall scheduling, leaving local scheduling to the client where the job resides.

[0099] (4) Data processing performance.

[0100] Similar to databases, Hadoop may require specialized optimization engineers to tune it according to the actual application needs; this is sometimes referred to as "Hadoop Performance Optimization" (HPO).

[0101] In this embodiment of the log storage and retrieval method, the storage compression rate can reach within 5% of the original size, and the average response time for 100 concurrent search requests can be controlled within 500 milliseconds. Compared with the commonly used ELK technology, it can save 90% of CPU, memory, and storage resource consumption. That is, under the same log volume and the same hardware configuration, the log read and write method in this embodiment can save 90% of the cost compared with the log storage and retrieval methods using ELK and other technologies. It is worth noting that this embodiment uses a simple index, that is, only type and time indexes, without word segmentation and inverted indexes. This embodiment uses parallel compression writing, whose write performance is more than 10 times that of the usual word segmentation and inverted index methods, and the storage space consumption is 1 / 10 of the usual word segmentation and inverted index methods. At the same time, the query performance is consistent with the usual word segmentation and inverted index methods. Through near real-time decompression retrieval, the search speed is optimized, and the number of disk I / O operations is greatly reduced. The search performance achieves millisecond-level response, which can meet most search needs.

[0102] In addition, all modules in this embodiment are HA architecture, using dual-replica storage to ensure data consistency and reliability, and support online horizontal scaling. There are no fatal single points of failure or problems with the inability to expand the space capacity. Moreover, no special hardware configuration is required, and no SSD solid-state storage disk is needed. It can run on ordinary servers (HDD storage) or virtual machines, which can effectively reduce hardware procurement costs.

[0103] Based on this, the log storage and retrieval method in this embodiment may include:

[0104] Step S10: Receive log data and compress the log data in parallel and write it to a preset disk space;

[0105] It should be noted that in this embodiment, the TCP protocol is used for log sending and receiving. The log sender can format the log data into a space-separated string: type, application name, timestamp, log text\n, for example, app HelloWorld 2022011114311050742649Nice to meet you.\n. The type and application name can be changed to the required strings according to the actual situation for categorization.

[0106] After receiving log data in string format from the sender, the terminal device further compresses the log data in parallel and writes it to a preset disk space.

[0107] In this embodiment, parallel compressed storage is used, which makes the write performance more than 10 times that of the usual method, while the storage space consumption is 1 / 10 of the usual method, which greatly saves storage space and thus saves storage costs.

[0108] Step S20: Receive the search text sent by the user and determine the target search text corresponding to the search text;

[0109] After compressing and writing the log data in the memory cache to the preset disk space, the terminal device checks whether it has received a search request triggered by the user and the corresponding search text. If the search text sent by the user is detected, in order to match the log data that matches the search text, the terminal device needs to perform relevant preprocessing operations on the search text to obtain the corresponding target search text, so as to query the log data that matches the target search text.

[0110] Step S30: Determine the list of files to be searched from the disk space according to the preset search time range, decompress and read the target log data that matches the target search text from the list of files to be searched, and store the target log data into the preset result set.

[0111] After writing log data from the memory cache to a preset disk space and receiving the target search text corresponding to the search text sent by the user, the terminal device will determine the list of files to be searched from the disk space according to the search time range determined by the user, and then match the target search text with the list of files to be searched to determine the log data that matches the target search text. The log data will be stored in a preset result set and returned to the corresponding search requester.

[0112] It should be noted that in this embodiment, based on the search time range predetermined by the user, a list of files to be searched is determined from the disk space using a time indexing method to respond to the search request that is triggered, which effectively improves the efficiency of query retrieval.

[0113] In this embodiment, after receiving log data sent as a string by the sender, the terminal device first writes the log data to a preset disk space using parallel compression. After compressing and writing the log data in the memory cache to the preset disk space, the terminal device checks whether it has received a user-triggered search request and corresponding search text. If a user-sent search text is detected, in order to match log data consistent with the search text, the terminal device needs to preprocess the search text to obtain a target search text that meets the retrieval criteria, so as to retrieve log data that matches the target search text. After writing the log data in the memory cache to the preset disk space and receiving the target search text corresponding to the user-sent search text, the terminal device determines a list of files to be searched from the disk space according to the search time range determined by the user, and then matches the target search text with the list of files to be searched to determine the log data consistent with the target search text. This log data is then stored in a preset result set, which is then returned to the corresponding search requester.

[0114] Compared to existing storage and retrieval methods, this invention compresses and writes received log data to a pre-defined disk space, identifies log data from this disk space that matches the search text sent by the user, and sends this log data to the corresponding search request initiator. Therefore, this invention achieves near real-time parallel compression and writing, saving significant storage space and thus drastically reducing disk I / O operations. Furthermore, this invention utilizes a time-based indexing method to achieve near real-time decompression retrieval and millisecond-level response, greatly improving search speed, similarly significantly reducing disk I / O operations, and meeting the vast majority of search requirements.

[0115] Based on this, the present invention does not require special hardware configuration or SSD solid-state storage disks, and can run on ordinary servers (HDD storage) or virtual machines, thus effectively reducing hardware procurement costs.

[0116] Furthermore, based on the first embodiment of the log storage and retrieval method of the present invention, a second embodiment of the log storage and retrieval method is proposed.

[0117] In this embodiment, step S10 above, "compressing and writing the log data in parallel to a preset disk space," may include:

[0118] Step S101: The log data is temporarily stored in a memory cache, and the log data in the memory cache is compressed and written in parallel to a preset disk space.

[0119] After receiving log data sent by the sender in string format, the terminal device first temporarily stores the log data in a memory buffer to perform unified parallel compression processing on all log data in the memory buffer. When the preset write conditions are met, the log data in the memory buffer is written to the preset disk space using parallel compression.

[0120] Furthermore, in step S101 above, "temporarily storing the log data in a memory cache and compressing and writing the log data in the memory cache into a preset disk space in parallel" may include:

[0121] Step S1011: The log parsing module receives log data, parses the log data to obtain log data in string form, and sends the log data in string form to the log compression and storage module.

[0122] Step S1012: Extract the log type, application name and log timestamp from the log data in string form through the log compression and storage module, and temporarily store the log data in the memory cache according to the log type, application name and log timestamp.

[0123] It should be noted that in this embodiment, when sending log data, the sender converts the log data into a string format for transmission. The string includes: type, application name, timestamp, and log text, with each character separated by a space. For example:

[0124] app HelloWorld 2022011114311050742649Nice to meet you.\n;

[0125] Therefore, the terminal device receives log data in string format through the log parsing module, extracts the first three segments of the string to obtain the type, application name, and timestamp, constructs a log object according to the type and application name, and temporarily stores it in the memory cache. Based on this, the terminal device needs to record both the earliest and latest log times for subsequent compressed file naming, in the following format:

[0126]

[0127] for example,

[0128]

[0129] Furthermore, the log data can be cached and compressed and written using the log compression and storage module, following the construction and naming conventions of the log data strings described above.

[0130] It should be noted that, in this embodiment, as Figure 3 As shown, the log parsing module (i.e., the Parser module) and the log compression and storage module (i.e., the Receiver module) store log data on disk. The query module (i.e., the Searcher module) and the gateway module (i.e., the Gateway module) search for target log data matching the user-triggered search text and return the target log to the user. Specifically, the Parser module is responsible for concurrently subscribing to the log data stream (Kafkatopic), parsing it into string data, and sending it to the Receiver module. The Parser module and the Receiver module have a many-to-one connection, allowing for horizontal scaling of instances to increase parsing and transmission capabilities.

[0131] Specifically, for example, the storage and retrieval system architecture is as follows: Figure 4 As shown, the log storage and search system consists of four modules:

[0132] Parser module: Multiple Parser groups concurrently subscribe to log data streams (Kafka topic), parse them into string data, and send them to the Receiver;

[0133] Receiver module: The Receiver caches log data in a memory cache and simultaneously synchronizes data to standby Receiver nodes (Receiver'). When the write condition is met, the memory cached data is compressed and written to disk.

[0134] Gateway module: Gateway is responsible for receiving user search requests and search text online, forwarding them to the corresponding Searcher group, aggregating the sharded search data returned by the Searcher group, sorting it, and returning it to the user;

[0135] Searcher module: Multiple Searcher groups are responsible for receiving search requests forwarded by the Gateway, retrieving data from disk (decompressed) and memory, and returning it to the Gateway.

[0136] Step S1013: When the number of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed at a preset refresh frequency, the log data in the memory cache is pushed to a preset compression queue through the log compression storage module.

[0137] Step S1014: Compress and write the log data in the preset compression queue to the disk space in parallel according to the preset log writing rules.

[0138] After the terminal device temporarily stores the log data as a string in the memory buffer through the Receiver module, when the pre-write conditions are met, the Receiver module directly pushes the string that meets the conditions in the memory buffer into the preset compression queue. Then, according to the preset log compression rules, the compression algorithm compresses the log data in the compression queue in parallel and writes it to the disk space.

[0139] It should be noted that, in this embodiment, the preset log writing rules include: the path and format of the log data when compressed and written to disk space, including: name / type / date / earliest log time_latest log time_number of logs.gz, for example:

[0140] / data / HelloWorld / app / 202201111431 / 2022011114311050345612_2022011114311050742649_2.gz.

[0141] Specifically, for example, when the number of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed according to a preset refresh frequency (for example, setting it to refresh every 30 seconds, and writing logs cached for more than 30 seconds to storage), the compression writing process is started, and cached objects that meet the refresh conditions are pushed into the compression queue. Then, gzip, 7zip, zstd or other compression algorithms can be used to compress the log data in the compression queue and write it to a file according to "application name / type / date / earliest log time_latest log time_number of logs.gz". After the compression writing is completed, the compression queue is cleared.

[0142] It should be noted that, in this embodiment, the Receiver module is responsible for receiving log parsing data and compressing and storing it to the file system on disk space. Furthermore, the Receiver module structure is as follows: Figure 5As shown, to ensure high availability, the Receiver is designed to run in pairs, consisting of two nodes, Receiver and Receiver' (Receiver' is the slave node of Receiver), each with a Disk Volume mounted to achieve dual-copy data storage. The storage of Receiver can be local storage or external storage.

[0143] For example, a Receiver receives parsed data from a Parser and simultaneously copies the data to Receiver'. The Receiver categorizes and stores the data in its own memory cache. When the write conditions (size, time) are met, the buffered data is compressed and written to disk. If a Receiver in a group fails, it disconnects from the Parser, stops receiving data, and waits for the Receiver to recover. If the Receiver recovers, Receiver' copies the memory data back to the Receiver in reverse synchronization. After data synchronization is complete, normal data reception and processing services resume (entering read-only mode and immediately flushing to disk during a failure). Alternatively, if a Receiver in a group fails, the Receiver will actively pause receiving data and wait for Receiver' to recover. If Receiver' recovers, the Receiver automatically copies the memory data back to Receiver' in synchronization. After data synchronization is complete, normal data reception and processing services resume.

[0144] Therefore, the Receiver module workflow includes:

[0145] (1) The Receiver receives the parsed data from the Parser and copies a copy of the data to the Receiver. The Receiver categorizes and stores the data in its own memory buffer. When the conditions for writing to disk are met (size, time), the buffer data is compressed and written to the disk.

[0146] (2) If the Receiver fails, Receiver' will take over the work of the Receiver, only accept query requests, and wait for the Receiver to recover. When the Receiver recovers, Receiver' will pause subscription and copy the memory data to the Receiver. After the Receiver completes data synchronization, it will resume normal operation (entering read-only mode and immediately flushing to disk in case of failure).

[0147] Furthermore, before step S10, "receiving log data and temporarily storing the log data in a memory cache," the method further includes:

[0148] Step S40: Establish a connection between the log parsing module and the log compression and storage module through the gateway module.

[0149] It should be noted that in this embodiment, the terminal device needs to connect the log parsing module and the log compression and storage module in advance before receiving and storing the log data.

[0150] Specifically, for example, multiple Parser units can be started concurrently. Each Parser provides its own configuration information upon startup to request the IP address and port of its partner Receiver from the Gateway and connects to the Receiver. The configuration information is automatically obtained when the service starts for the first time. If obtained, it is cached in a local file, and the Parser attempts to update the registration information periodically. If the registration information cannot be obtained on the next startup, it will be read from the local cache file.

[0151] Based on this, the Parser module workflow includes:

[0152] (1) If the Receiver has already connected to the Parser and started subscribing to the log data stream (Kafka topic), it will parse the JSON structured log data into a standard string structure and send it to the Receiver;

[0153] (2) If the Receiver connection is lost, the Parser should try to re-establish the connection with the Receiver every once in a while. If the Receiver is a K8s container instance, the Parser should obtain the IP address and port of the partner Receiver instance from the Gateway. Once the connection is established, jump to step (1).

[0154] (3) If the Parser unit exits normally, it should send the last piece of parsed data before exiting; if the Parser unit exits unexpectedly, no more than 100 log data should be lost.

[0155] In this embodiment, the terminal device receives log data in string format through the log parsing module. It extracts the first three segments of the string to obtain the type, application name, and timestamp. A log object is then constructed according to the type and application name and temporarily stored in a memory cache. After the terminal device temporarily stores the log data as a string in the memory cache through the Receiver module, when the pre-write conditions are met, the Receiver module directly pushes the string that meets the conditions from the memory cache into a preset compression queue. Then, according to preset log compression rules, the log data in the compression queue is compressed in parallel and written to disk space using a compression algorithm.

[0156] In this invention, when storing log data, the type, application name, and timestamp of the corresponding string are extracted to construct a log object. This log object is then temporarily stored in a memory cache. Log data in the memory cache that meets the write conditions is pushed into a compression queue, and then the log data in the compression queue is written to disk space using parallel compression. Therefore, this invention improves the write performance of log storage through parallel compression, enabling near real-time processing of tens of millions of TPS log data from applications. It also significantly reduces the cost of storing massive amounts of logs and greatly saves storage space, consuming only 1 / 10 of existing related technologies. Furthermore, this invention does not require special hardware configurations or SSD solid-state storage disks and can run on ordinary servers (HDD storage) or virtual machines, effectively reducing hardware procurement costs.

[0157] Furthermore, based on the first and second embodiments of the log storage and retrieval method of the present invention, a third embodiment of the log storage and retrieval method of the present invention is proposed.

[0158] In this embodiment, step S20 above, "receiving the search text sent by the user and determining the target search text corresponding to the search text," may include:

[0159] Step S201: Receive the search text sent by the user through the gateway module, and perform text syntax verification and text operator recognition on the search text;

[0160] Step S202: Construct a search tree and logical matching string for the search text after text syntax verification and text operator recognition to obtain the target search text.

[0161] It should be noted that, in this embodiment, the terminal device receives the search text sent by the user through the Gateway module, and performs preprocessing operations on the search text including: text syntax verification, text operator recognition, search tree construction and logical matching string construction to obtain the target search text that meets the retrieval criteria, so as to match the corresponding target log data based on the target search text.

[0162] Specifically, for example, the retrieval method in this embodiment uses string matching, similar to the grep command in Linux or the Ctrl+F function in a text editor. It can distinguish between uppercase and lowercase matching, exact matching, and fuzzy matching, supports logical operations such as AND, OR, and NOT, and supports numerical comparisons (e.g., code > 200). Preprocessing operations include:

[0163] (1) Validate the text syntax and exclude illegal search text, such as checking for illegal characters and missing quotation marks;

[0164] (2) Search text for tokens, identify logical and comparison operators. For example, "Nice to AND you" will be tokenized into an array ["Nice to", "AND", "you"];

[0165] (3) Construct a vocabulary search tree, using a data structure similar to a B+ tree, for example, ... Figure 5 As shown, the array ["Niceto", "AND", "you"] is constructed as follows: Figure 5 The search tree shown;

[0166] (4) Establish logical matching strings, replace the original word positions with logical operators and logical values, and establish mapping relationships. For example, convert the array ["Nice to", "AND", "you"] into the string "false&&false".

[0167] After text syntax validation, text operator recognition, search tree construction, and logical matching string construction, the target search text corresponding to the search text will be obtained. The target search text will then be sent to the Searcher module, which will determine the target log data that matches the target search text.

[0168] It should be noted that, in this embodiment, as Figure 6 As shown, Gateway provides log query gateway services and registration information services for log storage and retrieval nodes, while Searcher retrieves results from compressed log files based on conditions.

[0169] In addition, to ensure high availability, the Gateway requires at least two nodes running on the backend of Nginx. The Searcher and Receiver have a one-to-two connection relationship, with each Searcher mounting two storage blocks corresponding to the Receiver group. The Gateway distributes the search request to the Searcher based on the search text triggered by the user. The Searcher retrieves the stored data and the memory data of the Receiver or Receiver' and returns the results to the Gateway. The Gateway then merges and sorts the results and returns them to Nginx.

[0170] It's worth noting that the connection between Gateway and Receiver is a one-to-many relationship, such as a 1:2 connection, where each Searcher mounts two storage blocks corresponding to the Receiver group. Gateway instances can be horizontally scaled to increase request processing capacity. To ensure high availability, Gateway requires at least two nodes, running behind services with load balancing capabilities such as Nginx or HA proxy. The Searcher and Receiver have a 1:2 connection, with each Searcher mounting two storage blocks corresponding to the Receiver group. Searchers run behind Gateway, and Searcher instances can be horizontally scaled to increase request processing capacity.

[0171] Based on this, the Searcher workflow is as follows:

[0172] When the Searcher receives a query request forwarded by the Gateway, it starts a concurrent retrieval thread to search the stored data according to the query conditions and returns the results to the Gateway. If no results are found, it returns an empty set.

[0173] The Gateway workflow is as follows:

[0174] (1) After the Gateway starts, it listens for and receives node registration information. If the Parser or Receiver starts in K8s mode, the Gateway actively obtains the instance name and instance IP address information of the Parser or Receiver and registers it.

[0175] (2) When the Gateway receives a partner node IP request, it queries and returns the partner node IP and port information according to the partner node association policy;

[0176] (3) When the Gateway receives the user's search request and search text, it distributes the query request to the Searcher based on the search text. The Searcher concurrently retrieves the stored data and returns the results to the Gateway. Finally, the Gateway merges and sorts the results and returns them to the upstream.

[0177] Furthermore, in step S30 above, the step of "determining the list of files to be searched from the disk space according to a preset search time range" includes:

[0178] Step S301: Determine the time range of the logs based on the file names of the logs in the disk space, and obtain the target time range by performing an intersection operation between the time range and the preset search time range through the query module.

[0179] Step S302: Obtain the log data corresponding to the target time range from the disk space to obtain the list of files to be searched.

[0180] It should be noted that, in this embodiment, the preset time search range is the time search range set by the user when performing log retrieval. Based on this, the terminal device will determine the list of files to be searched from the log data in the disk space according to this time search range.

[0181] Specifically, for example, the terminal device only needs to select log files within the time search range determined by the user. It's worth noting that when compressing and naming the files, a compression format and path of "application name / type / date / earliest log time_latest log time_number of logs.gz" is used, recording the earliest and latest log times for the compressed file name. Based on this, the terminal device can determine the time range of the logs within the file by parsing the filename, and then perform an intersection operation with the search time range to obtain the target time range. It can then retrieve the log data corresponding to that target time range, ultimately obtaining a list of files to be searched within the target time range.

[0182] Further, in step S30 above, "reading target log data matching the target search text from the list of files to be searched" may include:

[0183] Step S303: Through the query module, the data character stream is obtained by multi-threaded parallel decompression of the file list to be searched, and the data character stream is stored in the corresponding cache.

[0184] Step S304: Determine the data character stream in the cache that has not exceeded the preset search time range, and read the data character stream sequentially;

[0185] Step S305: Compare the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file;

[0186] Step S306: If the currently read character matches any character in the character group, the currently read character is temporarily stored in a preset log cache, and the preset pointer is pointed to the next character in the character group that matches the currently read character.

[0187] Step S307: Determine the target word in the log cache, and when all characters in the target word are matched correctly, perform a logical operation on the logical matching string according to the logical value of the target word in the corresponding logical matching string, so as to obtain the target log data when the result of the logical operation is true, and clear the log cache.

[0188] After step S305 above, "comparing the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file", the following may also be included:

[0189] Step S308: If the currently read character fails to match any of the characters in the character group, then the preset pointer is reset.

[0190] It should be noted that in this embodiment, the Searcher module uses concurrent multi-threading to decompress and read the list of files to be searched. Furthermore, since a search tree has been pre-built for the target search files, the terminal device uses this search tree to match each line of log text in the search file list. The specific steps include:

[0191] (1) Obtain a data character stream by decompressing a file in the list of files to be searched by each thread, and write the data character stream into a fixed-size buffer. For example, each time 1024 bytes of data are decompressed and read, the 1024 bytes are stored in a buffer of size 1024 bytes.

[0192] (2) Extract the cached time portion and perform time range matching. Only read the data character stream in the cache that does not exceed the preset search time range sequentially. For logs that exceed the search time range, read the newline character directly without performing word segmentation matching and jump to step (1). Otherwise, proceed to step (3).

[0193] (3) Read the data character stream in the cache character by character sequentially, using a preset pointer `current` to point to the current matching character array in the search tree (the pointer points to the root of the search tree by default). For each character read, match it against the character array of the pointer `current`. If a match is found (hit), the pointer `current` will point to the next character in the character array; otherwise, the pointer `current` will be reset, i.e., pointed to the root. If the last character of a word has been matched, change the logical value of the corresponding position in the logical matching string to true.

[0194] (4) Store the read characters into a fixed-size log cache, with the log cache size set to twice the maximum length of a single log entry. If a newline character is encountered, it indicates that a log entry has ended. Then, perform logical operations on the logically matched strings. If the operation result is true, obtain the target log data corresponding to the string, store the target log data in the result set, and then clear the log cache.

[0195] (5) Repeat steps (1)-(4) until the list of files to be searched is finished;

[0196] It should be noted that in this embodiment, each thread executes steps (1)-(4) until the number of results in the result set reaches the number threshold or the search of all files in the file list to be searched has been completed.

[0197] Furthermore, after step S40 above, "storing the target log data into a preset result set", the method further includes:

[0198] Step S50: The gateway module merges and summarizes the preset result set in chronological order, returns the merged and summarized search results to the Nginx server mounted on the gateway module, and ends the search task.

[0199] After the terminal device saves the target log data that matches the search text triggered by the user to the preset result set, it waits for all search threads to finish, then uses the Gateway module to aggregate the result sets of all threads, sorts them in chronological order, returns the results to the load-balancing servers such as Nginx and HA proxy mounted on the Gateway module, and ends the current search task.

[0200] In this embodiment, upon receiving the search text sent by the user, preprocessing operations including text syntax verification, text operator recognition, search tree construction, and logical matching string construction are performed to obtain target search text that meets the retrieval criteria. Based on this target search text, corresponding target log data is matched. The terminal device only needs to select log files within the time search range determined by the user. It is worth noting that, during file compression and naming, the earliest and latest log times are recorded using the compression format "application name / type / date / earliest log time_latest log time_number of logs.gz". Based on this, the terminal device can determine the time range of the logs within the file through filename parsing, and perform an intersection operation with the search time range to obtain the target time range. The corresponding log data within this target time range is then retrieved, ultimately resulting in a list of files to be searched within the target time range. The system employs concurrent multi-threading to decompress and read the list of files to be searched. Since a search tree has been pre-built for the target search files, the terminal device uses this search tree to match each line of log text in the search file list to obtain a result set. The result set is then merged and summarized in chronological order, and the merged and summarized search results are returned to the user, thus ending the search task.

[0201] Therefore, in this invention, based on log type and time indexing method, the Searcher and Gateway modules achieve fast full-text log retrieval through parallel decompression, achieving millisecond-level response times. This meets most search requirements, achieving near real-time search performance while significantly reducing disk I / O operations and effectively improving query efficiency. Furthermore, this invention requires no special hardware configuration or SSD solid-state storage, and can run on ordinary servers (HDD storage) or virtual machines, effectively reducing hardware procurement costs.

[0202] Furthermore, embodiments of the present invention also propose a log storage and retrieval system, referring to... Figure 8 , Figure 8 This is a schematic diagram of the functional modules of an embodiment of the log storage and retrieval of the present invention. Figure 8 As shown, the log storage and retrieval system of the present invention includes:

[0203] A compression writing module is used to receive log data and compress the log data in parallel and write it to a preset disk space;

[0204] The receiving module is used to receive the search text sent by the user and determine the target search text corresponding to the search text;

[0205] The retrieval module is used to determine a list of files to be searched from the disk space according to a preset search time range, decompress and read target log data that matches the target search text from the list of files to be searched, and store the target log data into a preset result set.

[0206] Furthermore, the storage module 10 includes:

[0207] The compression writing unit is used to temporarily store the log data in a memory cache and compress the log data in the memory cache in parallel and write it to a preset disk space.

[0208] Furthermore, the compression writing unit includes:

[0209] The parsing and sending unit is used to receive log data through the log parsing module, parse the log data to obtain log data in string form, and send the log data in string form to the log compression and storage module;

[0210] The first storage unit is used to extract the log type, application name and log timestamp from the log data in string form through the log compression storage module, and temporarily store the log data in the memory cache according to the log type, application name and log timestamp.

[0211] The push subunit is used to push the log data in the memory cache to a preset compression queue through the log compression storage module when the number of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed according to a preset refresh frequency.

[0212] The compression writing subunit is used to compress and write log data in the preset compression queue to the disk space in parallel according to preset log writing rules.

[0213] Furthermore, the log storage and retrieval system also includes:

[0214] The connection establishment module is used to establish a connection between the log parsing module and the log compression and storage module through the gateway module.

[0215] Further, the receiving module 20 includes:

[0216] The search text receiving unit is used to receive search text sent by the user through the gateway module, and to perform text syntax verification and text operator recognition on the search text.

[0217] The target search text construction unit is used to construct a search tree and logical matching string from the search text after text syntax verification and text operator recognition to obtain the target search text.

[0218] Furthermore, the retrieval module 30 includes:

[0219] The intersection operation unit is used to determine the time range of the logs based on the file names of the logs in the disk space, and to obtain the target time range by performing an intersection operation between the time range and the preset search time range through the query module.

[0220] The file list acquisition unit is used to obtain the file list to be searched by retrieving log data corresponding to the target time range from the disk space.

[0221] Furthermore, the retrieval module 30 includes:

[0222] The second storage unit is used to perform multi-threaded parallel decompression of the file list to be searched through the query module to obtain a data character stream, and store the data character stream into the corresponding cache;

[0223] A reading unit is used to determine the data character stream in the cache that has not exceeded the preset search time range, and to read the data character stream sequentially;

[0224] The comparison unit is used to compare the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file;

[0225] The third storage unit is used to temporarily store the currently read character in a preset log cache if the currently read character matches any character in the character group, and at the same time, to point the preset pointer to the next character in the character group that matches the currently read character;

[0226] A logic operation unit is used to determine the target word in the log cache, and when all characters in the target word are matched correctly, to perform a logic operation on the logic matching string according to the logic value of the target word in the corresponding logic matching string, so as to obtain the target log data when the result of the logic operation is true, and to clear the log cache;

[0227] The reset unit is used to reset the preset pointer if the currently read character fails to match any of the characters in the character group.

[0228] Furthermore, the log storage and retrieval system also includes:

[0229] The return unit is used to merge and summarize the preset result set in chronological order through the gateway module, return the merged and summarized search results to the Nginx server mounted on the gateway module, and end the search task.

[0230] The specific implementation methods of each functional module of the log storage and retrieval system of the present invention are basically the same as those of the above-described log storage and retrieval methods, and will not be repeated here.

[0231] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a log storage and retrieval program, wherein the log storage and retrieval program, when executed by a processor, implements the steps of the log storage and retrieval method described above.

[0232] The various embodiments of the log storage and retrieval system and computer-readable storage medium of the present invention can be referred to the various embodiments of the log storage and retrieval method of the present invention, and will not be repeated here.

[0233] Furthermore, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of the log storage and retrieval method as described in any of the embodiments of the log storage and retrieval method above.

[0234] The specific embodiments of the computer program product of the present invention are basically the same as the embodiments of the above-described log storage and retrieval method, and will not be described in detail here.

[0235] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0236] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0237] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0238] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for storing and retrieving logs, characterized in that, The storage and retrieval method includes the following steps: Log data is received through a log parsing module and compressed and written to a preset disk space in parallel through a log compression and storage module. The log parsing module and the log compression and storage module have a many-to-one connection. The log compression and storage module includes a master node and slave nodes. The master node receives the parsed data from the log parsing module and copies the data to the slave nodes. If the master node fails, the slave nodes will take over the work of the master node, only accepting query requests and waiting for the master node to recover. When the master node recovers, the slave nodes suspend subscriptions and copy the memory data to the master node. After the master node completes data synchronization, it resumes normal operation. The path and format of the compressed log data written to the disk space include the earliest log time and the latest log time. The earliest and latest log times are used for naming the compressed file. The gateway module receives the search text sent by the user and determines the target search text corresponding to the search text. The query module initiates a concurrent retrieval thread, determines a list of files to be searched from the disk space according to a preset search time range, decompresses and reads target log data matching the target search text from the list of files to be searched, and stores the target log data into a preset result set; the query module and the log compression storage module have a one-to-two connection relationship, with each query module mounting two storage blocks of the corresponding log compression storage module; the query module runs connected to the backend of the gateway module; The step of writing the log data to a preset disk space in parallel through the log compression storage module includes: The log parsing module receives log data, parses the log data to obtain log data in string format, and sends the log data in string format to the log compression and storage module. The log compression and storage module extracts the log type, application name, and log timestamp from the string-format log data, and categorizes and temporarily stores the log data in the memory cache according to the log type, application name, and log timestamp. When the amount of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed at a preset refresh frequency, the log data in the memory cache is pushed to a preset compression queue through the log compression storage module. Log data in the preset compression queue is written to the disk space in a parallel compression manner according to the preset log writing rules; The step of determining the list of files to be searched from the disk space according to a preset search time range includes: The time range of the logs is determined based on the file names of the logs in the disk space. The target time range is obtained by intersecting the time range with the preset search time range through the query module. Obtain the log data corresponding to the target time range from the disk space to obtain the list of files to be searched; The step of decompressing and reading target log data matching the target search text from the list of files to be searched includes: The query module performs multi-threaded parallel decompression on the list of files to be searched to obtain a data character stream, and stores the data character stream into the corresponding cache. Identify data character streams in the cache that are within the preset search time range, and read the data character streams sequentially. Compare the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file; If the currently read character matches any character in the character group, the currently read character is temporarily stored in a preset log cache, and the preset pointer is pointed to the next character in the character group that matches the currently read character. The target word in the log cache is determined, and when all characters in the target word are matched correctly, a logical operation is performed on the logical matching string according to the logical value of the target word in the corresponding logical matching string. When the result of the logical operation is true, the target log data is obtained, and the log cache is cleared. The matching process for the data character stream does not involve word segmentation and does not use an inverted index.

2. The log storage and retrieval method as described in claim 1, characterized in that, Before the steps of receiving log data through the log parsing module and writing the log data to a preset disk space in parallel through the log compression and storage module, the method further includes: The gateway module establishes a connection between the log parsing module and the log compression and storage module.

3. The log storage and retrieval method as described in claim 2, characterized in that, The step of receiving the search text sent by the user through the gateway module and determining the target search text corresponding to the search text includes: The gateway module receives search text sent by the user and performs text syntax verification and text operator recognition on the search text. The search tree and logical matching strings are constructed from the search text after text syntax verification and text operator recognition to obtain the target search text.

4. The log storage and retrieval method as described in claim 1, characterized in that, After the step of comparing the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file, the method further includes: If the currently read character fails to match any of the characters in the character group, the preset pointer is reset.

5. The log storage and retrieval method as described in claim 2 or 3, characterized in that, After the step of storing the target log data into a preset result set, the method further includes: The gateway module merges and summarizes the preset result set in chronological order, returns the merged and summarized search results to the Nginx server mounted on the gateway module, and ends the search task.

6. A log storage and retrieval system, characterized in that, The log storage and retrieval system includes: The compression and writing module receives log data through the log parsing module and compresses and writes the log data in parallel to a preset disk space through the log compression and storage module. The log parsing module and the log compression and storage module have a many-to-one connection relationship. The log compression and storage module includes a master node and slave nodes. The master node receives the parsed data from the log parsing module and copies a copy of the data to the slave nodes. If the master node fails, the slave nodes will take over the work of the master node, only accepting query requests and waiting for the master node to recover. When the master node recovers, the slave nodes suspend subscriptions and copy the memory data to the master node. After the master node completes data synchronization, it resumes normal operation. The path and format of the log data compression and writing to the disk space include the earliest log time and the latest log time. The earliest log time and the latest log time are used for naming the compressed file. The receiving module is used to receive the search text sent by the user through the gateway module and determine the target search text corresponding to the search text. The retrieval module is used to initiate concurrent retrieval threads through the query module, determine a list of files to be searched from the disk space according to a preset search time range, decompress and read target log data matching the target search text from the list of files to be searched, and store the target log data into a preset result set; the query module and the log compression storage module have a one-to-two connection relationship, with each query module mounting two storage blocks of the corresponding log compression storage module; the query module runs connected to the backend of the gateway module; The step of compressing and writing the log data in parallel to a preset disk space via the log compression storage module includes: The log parsing module receives log data, parses the log data to obtain log data in string format, and sends the log data in string format to the log compression and storage module. The log compression and storage module extracts the log type, application name, and log timestamp from the string-format log data, and categorizes and temporarily stores the log data in the memory cache according to the log type, application name, and log timestamp. When the amount of log data in the memory cache reaches a preset threshold or when the memory cache is refreshed at a preset refresh frequency, the log data in the memory cache is pushed to a preset compression queue through the log compression storage module. Log data in the preset compression queue is written to the disk space in a parallel compression manner according to the preset log writing rules; The step of determining the list of files to be searched from the disk space according to a preset search time range includes: The time range of the logs is determined based on the file names of the logs in the disk space. The target time range is obtained by intersecting the time range with the preset search time range through the query module. Obtain the log data corresponding to the target time range from the disk space to obtain the list of files to be searched; The step of decompressing and reading target log data matching the target search text from the list of files to be searched includes: The query module performs multi-threaded parallel decompression on the list of files to be searched to obtain a data character stream, and stores the data character stream into the corresponding cache. Identify data character streams in the cache that are within the preset search time range, and read the data character streams sequentially. Compare the currently read character with the character group pointed to by the preset pointer in the corresponding search tree of the target search file; If the currently read character matches any character in the character group, the currently read character is temporarily stored in a preset log cache, and the preset pointer is pointed to the next character in the character group that matches the currently read character. The target word in the log cache is determined, and when all characters in the target word are matched correctly, a logical operation is performed on the logical matching string according to the logical value of the target word in the corresponding logical matching string. When the result of the logical operation is true, the target log data is obtained, and the log cache is cleared. In this process, the pattern of the data character stream is not segmented and does not use an inverted index.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a log storage and retrieval program stored in the memory and executable on the processor. When the log storage and retrieval program is executed by the processor, it implements the steps of the log storage and retrieval method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a log storage and retrieval program, which, when executed by a processor, implements the steps of the log storage and retrieval method as described in any one of claims 1 to 5.

Images