A log retrieval method and device based on distributed storage and a storage medium

Abstract

The application discloses a kind of log retrieval method, device and storage medium based on distributed storage, the log retrieval method includes: log is classified according to self-defined category division condition, and according to the set time dimension is split into the log block corresponding to each category;Each log block is stored and the log block index is established;Target log retrieval information is received, the target log block corresponding to target log is retrieved by log block index information, and the retrieval result information of target log is obtained by full-text retrieval to target log block.The application indexes log block, not each log, so the overall index data volume will be reduced by more than 1000 times, and there is no longer need to update index when writing log, which saves a lot of CPU;Correspondingly, when log retrieval is carried out, the log block to which the target retrieval log belongs is retrieved first, and then full-text retrieval is carried out in the log block, which greatly improves the efficiency of retrieval.

Description

Technical Field

[0001] This invention relates to the field of log retrieval technology, specifically to a log retrieval method, apparatus, and storage medium based on distributed storage. Background Technology

[0002] In a cloud computing environment, a large number of physical devices and business systems are deployed in a distributed manner. At the same time, platform monitoring and management systems for operation and maintenance, security management, etc. are also deployed. Log information has become an important means of platform monitoring and management. Massive amounts of distributed log information from different devices and systems need to be used by multiple systems. Once a device in the platform has a problem, log information becomes an important means of troubleshooting and preventing faults.

[0003] In existing technologies, most log search is achieved by building an ELK platform (a distributed log search platform consisting of an Elasticsearch search server, a Logstash data collection engine, and a Kibana visualization platform). Elasticsearch, based on the Lucene search engine (a full-text search engine), provides index storage, cluster management, and search functions; Logstash is responsible for log collection; and Kibana provides the log query interface. Specifically, Logstash collects logs and stores them uniformly in the Elasticsearch cluster. When a user initiates a log search request through Kibana, the corresponding logs are searched in the cluster and displayed in Kibana.

[0004] However, the ELK platform requires full-text indexing of all logs, and both the index and log data are stored on the local disk. Log scenarios will face the following two situations:

[0005] 1. The data volume is huge. A single user request will be processed through multiple services, resulting in hundreds of log entries. Therefore, the entire cloud computing environment will generate a massive amount of logs that need to be stored, creating enormous storage pressure for such a large volume of logs.

[0006] 2. Long retention time: Logs need to be retained for a long time for subsequent troubleshooting and auditing.

[0007] When faced with massive logs, using the ELK platform puts a lot of pressure on the machine's CPU and disk space. Storing a large number of logs and providing retrieval services undoubtedly requires high costs.

[0008] In view of this, the present invention is hereby proposed. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention proposes a log retrieval method, apparatus, and storage medium based on distributed storage. It aims to reduce the cost of log retrieval platforms in scenarios with massive log volumes, enabling the use of fewer CPUs to support a larger volume of log writes, thereby reducing the storage cost per GB by an order of magnitude. The specific technical solution is as follows:

[0010] A log retrieval method based on distributed storage includes:

[0011] The logs are categorized according to custom classification criteria and then split into log blocks corresponding to each category based on the set time dimension.

[0012] Storing and creating log block indexes for each log block;

[0013] Receive target log retrieval information, retrieve the target log block corresponding to the target log through the log block index information, and perform a full-text search on the target log block to obtain the retrieval results information of the target log.

[0014] As an optional embodiment of the present invention, the step of classifying logs according to custom category division conditions and splitting them into log blocks corresponding to each category according to a set time dimension includes:

[0015] Custom criteria for classifying logs into the same category include: whether they are logs generated by the same machine, and / or whether they belong to the same module, and / or whether they are logs of the same type, or whether they are logs at the same level.

[0016] As an optional embodiment of the present invention, the log retrieval method based on distributed storage of the present invention deploys log collection agents on each machine. The log collection agents obtain log information of the deployed machines and attach the machine name tag, and / or module name tag, and / or log type tag, and / or log level tag when uploading the log information.

[0017] Optionally, the module name includes the names of the various services deployed on the machine;

[0018] Optionally, the log types include business access logs (access.log), and / or database access logs (sql.log), and / or business logs (server.log); the log types also support business customization, and the log collection agent will attach the log type to the log information for selection of the corresponding log type during retrieval;

[0019] Optionally, the log levels include:

[0020] DEBUG points out that fine-grained informational events are very helpful for debugging applications;

[0021] INFO indicates that the message highlights the application's execution process at a coarse-grained level;

[0022] A WARN message indicates a potential error.

[0023] ERROR indicates that although an error event has occurred, it does not affect the continued operation of the system;

[0024] FATAL indicates that each critical error event will cause the application to exit. As an optional embodiment of the present invention, the log retrieval method based on distributed storage includes:

[0025] A unified time dimension is set for all types of log blocks. After all collected logs are cached to the set unified time dimension, they are split into log blocks of the corresponding categories.

[0026] As an optional embodiment of the present invention, the step of storing each log block and establishing a log block index includes:

[0027] Each log block is stored in multiple levels according to the system's preset time conditions;

[0028] Log blocks that meet the first preset time condition are stored on the local disk, log blocks that meet the second preset time condition are compressed and uploaded to the remote storage, and cold log block data that meet the third preset time condition are transferred from the remote storage to the archive storage for storage.

[0029] As an optional embodiment of the present invention, the step of storing each log block and establishing a log block index includes:

[0030] Multi-level storage is performed on each log block according to the system's preset storage capacity.

[0031] Log blocks smaller than the first preset storage limit are stored on the local disk. When the number of log blocks stored on the local disk exceeds the first preset storage limit, the log blocks stored on the local disk for a longer period than the first set value are compressed and uploaded to the remote storage for storage.

[0032] When the number of log blocks stored on the remote storage exceeds the second preset storage limit, cold log block data that has been stored on the remote storage for a longer period than the second set value will be transferred from the remote storage to the archive storage for storage.

[0033] As an optional embodiment of the present invention, the step of establishing a log block index for each log block includes:

[0034] An index is created for each log block, and the index content includes: time, log block metadata, and log block storage location; wherein, the log block metadata includes machine information and / or module information and / or type information and / or level information to which the log block belongs.

[0035] As an optional embodiment of the present invention, the step of retrieving the target log block corresponding to the target log through the log block index information and performing a full-text search on the target log block to obtain the retrieval result information of the target log includes:

[0036] When performing target log retrieval, extract key information from the acquired target log that corresponds to the log block classification category;

[0037] The key information obtained is used to retrieve the target log block and its storage location from the log block index information;

[0038] Download the target log blocks stored on remote storage or archive storage to the local disk, and then perform a full-text search on each log block using the search information of the target log as the search criteria. The search results are then summarized and sorted to obtain the search results information.

[0039] This invention also provides a log retrieval device based on distributed storage, comprising:

[0040] The log block classification module categorizes logs according to custom classification criteria and splits them into log blocks corresponding to each category based on a set time dimension.

[0041] The log block storage module stores individual log blocks.

[0042] The invention also includes a log block retrieval module, which establishes a log block index for the stored log blocks. This module receives target log retrieval information, retrieves the target log block corresponding to the target log using the log block index information, and performs a full-text search on the target log block to obtain the retrieval results. The invention further provides a storage medium storing a computer-executable program. When executed, the computer-executable program implements the log retrieval method based on distributed storage described in any of the above embodiments.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] In the log retrieval method based on distributed storage of this invention, the smallest unit of log storage is no longer storing logs one by one, but rather classifying logs into log blocks and storing them according to these blocks. In this distributed storage-based log retrieval method, logs of the same type are aggregated together and then split into log blocks according to the time dimension. A log block may contain several thousand or tens of thousands of logs of the same type. Logs of the same type can be defined as those generated under the same host, and / or belonging to the same module, and / or of the same log type, and / or of the same log level; these can be considered as the same type of log block.

[0045] The log retrieval method based on distributed storage of this invention creates an index for each log block. Using the log block index information, we can quickly find the required log block. Therefore, creating a block index for log blocks in this invention can greatly improve the efficiency of index creation.

[0046] The log retrieval method based on distributed storage of the present invention indexes log blocks rather than each log entry, so the overall index data volume is reduced by more than a thousand times. When writing logs, there is no need to update the index, which saves a lot of CPU.

[0047] When retrieving logs, the log retrieval method based on distributed storage of this invention first retrieves the location of the log block to be retrieved from the log block index and retrieves them from remote storage or archive storage back to the local disk. Then, it performs a full-text search on the log blocks concurrently to obtain the log information we need. Because the size of the log block is fixed, the speed of full-text search on a single log block is also very fast. Finally, the results of all log block searches are aggregated and returned to the user. Therefore, this invention first retrieves the log block corresponding to the target log and then performs a full-text search on the log block, which greatly improves the efficiency of the retrieval.

[0048] The log retrieval method based on distributed storage of the present invention uses hierarchical storage for log blocks and separates hot and cold data using remote storage and archive storage. Compared with storing all data on local disks, it can save a lot of disk costs. Attached image description:

[0049] Figure 1 A flowchart illustrating the hierarchical storage process for log blocks in the log retrieval method based on distributed storage according to an embodiment of the present invention;

[0050] Figure 2 The flow of the log retrieval method based on distributed storage in this embodiment of the invention Figure 1 ;

[0051] Figure 3 The flow of the log retrieval method based on distributed storage in this embodiment of the invention Figure 2 ;

[0052] Figure 4 The flow of the log retrieval method based on distributed storage in this embodiment of the invention Figure 3 ;

[0053] Figure 5 The flow of the log retrieval method based on distributed storage in this embodiment of the invention Figure 4 . Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0055] Therefore, the following detailed description of embodiments of the present invention is not intended to limit the scope of the claimed invention, but merely illustrates some embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0056] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0057] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0058] In the description of this invention, it should be noted that the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. These terms are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0059] See Figure 1 and Figure 2 As shown, this embodiment provides a log retrieval method based on distributed storage, including:

[0060] The logs are categorized according to custom classification criteria and then split into log blocks corresponding to each category based on the set time dimension.

[0061] Storing and creating log block indexes for each log block;

[0062] Receive target log retrieval information, retrieve the target log block corresponding to the target log through the log block index information, and perform a full-text search on the target log block to obtain the retrieval results information of the target log.

[0063] In this embodiment of the log retrieval method based on distributed storage, the smallest unit of log storage is no longer storing logs one by one. Instead, logs are classified and divided into log blocks and stored according to these blocks. In this distributed storage-based log retrieval method, logs of the same type are aggregated together and then split into log blocks according to the time dimension. A log block may contain several thousand or tens of thousands of logs of the same type. Logs of the same type can be defined as those generated under the same host, and / or belonging to the same module, and / or of the same log type, and / or at the same log level; these can be considered as the same type of log block.

[0064] The log retrieval method based on distributed storage in this embodiment creates an index for each log block. Using the log block index information, we can quickly find the required log block. Therefore, creating a block index for log blocks in this embodiment can greatly improve the efficiency of index creation.

[0065] The log retrieval method based on distributed storage in this embodiment indexes log blocks rather than each log entry, thus reducing the overall index data volume by more than a thousand times. Furthermore, it eliminates the need to update the index when writing logs, which saves a significant amount of CPU resources.

[0066] When retrieving logs, the log retrieval method based on distributed storage in this embodiment first retrieves the location of the log block to be retrieved from the log block index and retrieves them from remote storage or archive storage back to the local disk. Then, it performs a full-text search on the log blocks concurrently to obtain the log information we need. Because the size of the log block is fixed, the speed of full-text search on a single log block is also very fast. Finally, we aggregate the results of all log block searches and return them to the user. Therefore, this embodiment first retrieves the log block corresponding to the target log and then performs a full-text search on the log block, which greatly improves the efficiency of the retrieval.

[0067] As an optional implementation of this embodiment, in the log retrieval method based on distributed storage in this embodiment, the step of classifying logs according to custom category division conditions and splitting them into log blocks corresponding to each category according to a set time dimension includes:

[0068] Custom criteria for classifying logs into the same category include: whether they are logs generated by the same machine, and / or whether they belong to the same module, and / or whether they are logs of the same type, or whether they are logs at the same level.

[0069] The log retrieval method based on distributed storage in this embodiment can divide logs into log blocks by defining one of the above-mentioned division conditions, or by combining multiple division conditions to divide logs into multi-level blocks. Specifically:

[0070] For example, logs can be divided into first-level log blocks based on the machine or module that generated them, and then further divided into second-level log blocks based on log type or log level. When performing log retrieval, the first-level log blocks are retrieved first, then the second-level log blocks are retrieved from the first-level log blocks, and a full-text search is performed in the second-level log blocks to obtain the retrieval results of the target log.

[0071] The log retrieval method based on distributed storage in this embodiment deploys log collection agents on each machine. The log collection agents obtain log information from the deployed machines and attach the machine name tag, and / or module name tag, and / or log type tag, and / or log level tag when uploading the log information.

[0072] Optionally, the module name includes the names of the various services deployed on the machine; the services are all containerized, and multiple services are deployed on each machine, and the logs generated by each service can be defined as the logs of a module.

[0073] Optionally, the log types include business access logs (access.log), and / or database access logs (sql.log), and / or business logs (server.log); the log types also support business customization, and the log collection agent will attach the log type to the log information for selection of the corresponding log type during retrieval;

[0074] Application logs contain various error messages, warnings, or other events written by the application. These messages can provide high-level logical information relevant to specific use cases. Application logs include: stack traces of exceptions occurring in use cases; warning messages about slow response times from external systems; and information about whether use cases are triggered or completed. Each application log message should provide useful information to help understand the application's behavior.

[0075] Web or application server logs: Most enterprise applications are deployed to web servers (such as Apache Tomcat) or application servers (such as Wildfly or WebSphere Liberty). Their log files provide you with information about technical issues and the current state of the server.

[0076] Garbage Collector Log: The garbage collector provides automatic memory management. It tracks all objects instantiated in the JVM and deletes those that are no longer in use. This is a very complex task, and several different garbage collector implementations are available.

[0077] Generally, garbage collection in Java is very efficient. It's highly effective, and many applications use the default implementation and its standard configuration. However, if your application must handle high loads or uses a large number of complex object structures, its performance may degrade over time. This typically happens when the garbage collector needs to spend increasingly more time managing available memory. Garbage collector logs provide essential information for analyzing all garbage collector activity.

[0078] System logs: The operating system writes certain events to these logs. Other applications that are part of the deployment can also do this. Therefore, system logs are a good place to get an overview of external events.

[0079] If you are running an application on Windows, this is the event log. Windows event logs contain important hardware and software events categorized into application security, settings, system, and forwarding events.

[0080] Linux system logging on Linux: The Linux syslog service and its later versions provide a highly configurable logging system. It writes messages triggered by applications or the operating system to files or sends them to remote systems.

[0081] Optionally, the log levels include:

[0082] DEBUG points out that fine-grained informational events are very helpful for debugging applications;

[0083] INFO indicates that the message highlights the application's execution process at a coarse-grained level;

[0084] A WARN message indicates a potential error.

[0085] ERROR indicates that although an error event has occurred, it does not affect the continued operation of the system;

[0086] FATAL indicates that each critical error event will cause the application to exit. Further, the log retrieval method based on distributed storage in this embodiment includes:

[0087] A unified time dimension is set for all types of log blocks. After all collected logs are cached to the set unified time dimension, they are split into log blocks of the corresponding categories.

[0088] As an optional implementation method in this embodiment, see Figure 3 As shown, the log retrieval method based on distributed storage in this embodiment includes storing each log block and establishing a log block index, which includes:

[0089] Each log block is stored in multiple levels according to the system's preset time conditions;

[0090] Log blocks that meet the first preset time condition are stored on the local disk, log blocks that meet the second preset time condition are compressed and uploaded to the remote storage, and cold log block data that meet the third preset time condition are transferred from the remote storage to the archive storage for storage.

[0091] This embodiment adopts a multi-level storage approach for log blocks, wherein:

[0092] Remote storage: The storage medium for remote disks, which can be object storage or network file systems, is much cheaper than local disks.

[0093] Archive storage: a low-cost offline cold data storage method suitable for long-term backup of massive amounts of unstructured data, at a lower cost than remote storage.

[0094] Since logs are generally time-sensitive, the longer the log is stored, the lower the probability of it being retrieved. To avoid putting storage pressure on the local disk due to long-term log storage, this embodiment periodically packages and uploads the logs stored on the local disk to remote storage according to preset time parameters, and transfers the logs that have been stored on the remote storage for too long to archive storage. This greatly reduces the storage space requirements of the local disk and reduces the hardware cost of implementing the log retrieval method based on distributed storage in this embodiment.

[0095] Therefore, the log retrieval method based on distributed storage in this embodiment uses hierarchical storage for log blocks, and separates hot and cold data using remote storage and archive storage. Compared with storing all data on local disks, it can save a lot of disk costs.

[0096] As an optional implementation method in this embodiment, see Figure 4 As shown, the log retrieval method based on distributed storage in this embodiment includes storing each log block and establishing a log block index, which includes:

[0097] Multi-level storage is performed on each log block according to the system's preset storage capacity.

[0098] Log blocks smaller than the first preset storage limit are stored on the local disk. When the number of log blocks stored on the local disk exceeds the first preset storage limit, the log blocks stored on the local disk for a longer period than the first set value are compressed and uploaded to the remote storage for storage.

[0099] When the number of log blocks stored on the remote storage exceeds the second preset storage limit, cold log block data that has been stored on the remote storage for a longer period than the second set value will be transferred from the remote storage to the archive storage for storage.

[0100] Furthermore, this embodiment combines storage space parameters with storage time, and while ensuring the remaining storage space of the local disk and remote storage, gradually transfers the log block storage location according to the length of storage time.

[0101] As an optional implementation of this embodiment, the log retrieval method based on distributed storage in this embodiment includes the following steps: Establishing a log block index for each log block.

[0102] An index is created for each log block, and the index content includes: time, log block metadata, and log block storage location; wherein, the log block metadata includes machine information and / or module information and / or type information and / or level information to which the log block belongs.

[0103] In this embodiment, when building a block index for a log block, only time, log block metadata, and log block storage location need to be considered. Compared to building a full-text index for each log entry, this greatly improves the efficiency of index building. At the same time, when performing log retrieval, the log block to which the target log belongs is retrieved first, and then a full-text search is performed within the log block, which greatly improves the retrieval efficiency.

[0104] As an optional implementation method in this embodiment, see Figure 5 As shown in this embodiment, the log retrieval method based on distributed storage, when performing target log retrieval, involves retrieving the target log block corresponding to the target log through log block index information, and then performing a full-text search on the target log block to obtain the retrieval result information of the target log, including:

[0105] When performing target log retrieval, extract key information from the acquired target log that corresponds to the log block classification category;

[0106] The key information obtained is used to retrieve the target log block and its storage location from the log block index information;

[0107] Download the target log blocks stored on remote storage or archive storage to the local disk, and then perform a full-text search on each log block using the search information of the target log as the search criteria. The search results are then summarized and sorted to obtain the search results information.

[0108] This embodiment also provides a log retrieval device based on distributed storage, including:

[0109] - Classify logs according to custom category division criteria, and split them into log blocks corresponding to each category according to the set time dimension;

[0110] The log block storage module stores individual log blocks.

[0111] The system also includes a log block retrieval module, which establishes a log block index for the stored log blocks. The log block retrieval module receives target log retrieval information, retrieves the target log block corresponding to the target log through the log block index information, and performs a full-text search on the target log block to obtain the retrieval result information of the target log.

[0112] In the log retrieval device based on distributed storage in this embodiment, the smallest unit for storing logs in the log block storage module is no longer storing them one by one. Instead, logs are classified and divided into log blocks and stored according to these log blocks. In this log retrieval device based on distributed storage in this embodiment, the log block classification module aggregates logs of the same type together and splits them into log blocks according to the time dimension. A log block may contain several thousand or tens of thousands of logs of the same type. Logs of the same type can be defined as those generated under the same host, and / or belonging to the same module, and / or of the same log type, and / or of the same log level.

[0113] In the log retrieval device based on distributed storage in this embodiment, the log block retrieval module will create an index for each log block. By using the log block index information, we can quickly find the log block we need.

[0114] When retrieving logs, the log retrieval device based on distributed storage in this embodiment first retrieves the location of the log block to be retrieved from the log block index, and then retrieves them from the remote storage or archive storage of the log block storage module back to the local disk. Then, it concurrently performs full-text searches on the log blocks to obtain the required log information. Because the size of the log block is fixed, the speed of full-text search on a single log block is also very fast. Finally, the results of all log block searches are aggregated and returned to the user.

[0115] As an optional implementation of this embodiment, in the log retrieval device based on distributed storage in this embodiment, the log block classification module defines the classification conditions for logs of the same category as follows: whether they are logs generated by the same machine, and / or whether they belong to the same module, and / or whether they are logs of the same type, or whether they are at the same log level.

[0116] In this embodiment of the log retrieval device based on distributed storage, the log block classification module can define one of the above-mentioned partitioning conditions for log partitioning when dividing logs into log blocks, or it can combine multiple partitioning conditions to partition multi-level log blocks. Specifically:

[0117] For example, logs can be divided into first-level log blocks based on the machine or module that generated them, and then further divided into second-level log blocks based on log type or log level. When performing log retrieval, the first-level log blocks are retrieved first, then the second-level log blocks are retrieved from the first-level log blocks, and a full-text search is performed in the second-level log blocks to obtain the retrieval results of the target log.

[0118] Specifically, the log retrieval device based on distributed storage in this embodiment includes log collection agent modules deployed on various machines. The log collection agent modules acquire log information from the deployed machines and attach the machine name tag, and / or module name tag, and / or log type tag, and / or log level tag when uploading the log information to the log block classification module.

[0119] Optionally, the module name includes the names of the various services deployed on the machine.

[0120] Optionally, the log types include business access logs (access.log), and / or database access logs (sql.log), and / or business logs (server.log); the log types also support business-defined types, and the log collection agent will append the log type to the log information for selection during retrieval. The log types may further include:

[0121] Application logs contain various error messages, warnings, or other events written by the application. These messages can provide high-level logical information relevant to specific use cases. Application logs include: stack traces of exceptions occurring in use cases; warning messages about slow response times from external systems; and information about whether use cases are triggered or completed. Each application log message should provide useful information to help understand the application's behavior.

[0122] Web or application server logs: Most enterprise applications are deployed to web servers (such as Apache Tomcat) or application servers (such as Wildfly or WebSphere Liberty). Their log files provide you with information about technical issues and the current state of the server.

[0123] Garbage Collector Log: The garbage collector provides automatic memory management. It tracks all objects instantiated in the JVM and deletes those that are no longer in use. This is a very complex task, and several different garbage collector implementations are available.

[0124] Generally, garbage collection in Java is very efficient. It's highly effective, and many applications use the default implementation and its standard configuration. However, if your application must handle high loads or uses a large number of complex object structures, its performance may degrade over time. This typically happens when the garbage collector needs to spend increasingly more time managing available memory. Garbage collector logs provide essential information for analyzing all garbage collector activity.

[0125] System logs: The operating system writes certain events to these logs. Other applications that are part of the deployment can also do this. Therefore, system logs are a good place to get an overview of external events.

[0126] If you are running an application on Windows, this is the event log. Windows event logs contain important hardware and software events categorized into application security, settings, system, and forwarding events.

[0127] Linux system logging on Linux: The Linux syslog service and its later versions provide a highly configurable logging system. It writes messages triggered by applications or the operating system to files or sends them to remote systems.

[0128] Optionally, the log levels include:

[0129] DEBUG points out that fine-grained informational events are very helpful for debugging applications;

[0130] INFO indicates that the message highlights the application's execution process at a coarse-grained level;

[0131] A WARN message indicates a potential error.

[0132] ERROR indicates that although an error event has occurred, it does not affect the continued operation of the system;

[0133] FATAL indicates that each critical error event will cause the application to exit. Furthermore, in this embodiment's log retrieval device based on distributed storage, the log block classification module sets a unified time dimension for all categories of log blocks, caches all collected logs to the set unified time dimension, and then splits them into log blocks of the corresponding categories.

[0134] As an optional implementation of this embodiment, the log retrieval device based on distributed storage in this embodiment includes a log block storage module that performs multi-level storage for each log block according to system preset time conditions; log blocks that meet the first preset time condition are stored on the local disk, log blocks that meet the second preset time condition are compressed and uploaded to remote storage, and cold log block data that meet the third preset time condition are transferred from remote storage to archive storage for storage.

[0135] In this embodiment, the log block storage module adopts a multi-level storage method for log blocks, wherein:

[0136] Remote storage: The storage medium for remote disks, which can be object storage or network file systems, is much cheaper than local disks.

[0137] Archive storage: a low-cost offline cold data storage method suitable for long-term backup of massive amounts of unstructured data, at a lower cost than remote storage.

[0138] Since logs are generally time-sensitive, the longer the log is stored, the lower the probability of it being retrieved. To avoid putting storage pressure on the local disk due to long-term log storage, this embodiment periodically packages and uploads the logs stored on the local disk to remote storage according to preset time parameters, and transfers the logs that have been stored on the remote storage for too long to archive storage. This greatly reduces the storage space requirements of the local disk and reduces the hardware cost of implementing the log retrieval method based on distributed storage in this embodiment.

[0139] As an optional implementation of this embodiment, the log retrieval device based on distributed storage in this embodiment includes a log storage module that performs multi-level storage for each log block according to a system preset storage limit. Log blocks smaller than the first preset storage limit are stored on the local disk. When the number of log blocks stored on the local disk exceeds the first preset storage limit, log blocks whose storage time on the local disk exceeds a first set value are compressed and uploaded to remote storage for storage. When the number of log blocks stored on the remote storage exceeds the second preset storage limit, cold log block data whose storage time on the remote storage exceeds the second set value are transferred from the remote storage to archive storage for storage.

[0140] Furthermore, the log storage module in this embodiment combines storage space parameters with storage time, and while ensuring the remaining storage space of the local disk and remote storage, it gradually transfers the log block storage location according to the length of the storage time.

[0141] As an optional implementation of this embodiment, the log retrieval device based on distributed storage in this embodiment includes a log block retrieval module comprising:

[0142] The log block indexing module creates an index for each log block. The index content includes: time, log block metadata, and log block storage location. The log block metadata includes machine information and / or module information and / or type information and / or level information to which the log block belongs.

[0143] The log block indexing module in this embodiment only needs to consider time, log block metadata, and log block storage location when building a block index for a log block. Compared to building a full-text index for each log entry, it can greatly improve the efficiency of index building. At the same time, when performing log retrieval, it first retrieves the log block to which the target retrieval log belongs, and then performs a full-text search within the log block, which greatly improves the retrieval efficiency.

[0144] As an optional implementation of this embodiment, the log retrieval device based on distributed storage in this embodiment includes a log block retrieval module comprising a log block search module:

[0145] When performing target log retrieval, the log block search module retrieves the target log block corresponding to the target log using the log block index information, and performs a full-text search on the target log block to obtain the retrieval results information for the target log, including:

[0146] When performing target log retrieval, the log block search module extracts key information from the target log that corresponds to the log block classification category;

[0147] The log block search module retrieves the target log block and its location from the log block index information using the key information obtained.

[0148] The target log block, stored on remote storage or archive storage, is downloaded to the local disk. The log block search module then performs a full-text search on each log block using the search information of the target log as the search criteria, and summarizes and sorts the search results to obtain the search results information.

[0149] This embodiment also provides a storage medium storing a computer-executable program, which, when executed, implements the aforementioned method for retrieving massive logs based on distributed storage.

[0150] The storage medium described in this embodiment may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0151] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer-executable program, and when the computer program is executed by the processor, the processor executes the aforementioned method for retrieving massive logs based on distributed storage.

[0152] The electronic device is manifested in the form of a general-purpose computing device. It may contain one or more processors that work collaboratively. This invention also does not preclude distributed processing, meaning that processors may be distributed across different physical devices. The electronic device of this invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0153] The memory stores a computer-executable program, typically machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps of the method.

[0154] The memory includes volatile memory, such as random access memory (RAM) and / or cache memory, and may also be non-volatile memory, such as read-only memory (ROM).

[0155] It should be understood that the electronic device of the present invention may also include elements or components not shown in the examples above. For example, some electronic devices also include display units such as a display screen, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. Any electronic device capable of executing a computer-readable program in its memory to implement the method of the present invention or at least some steps of the method can be considered as an electronic device covered by the present invention.

[0156] From the above description of the embodiments, those skilled in the art will readily understand that the present invention can be implemented by hardware capable of executing specific computer programs, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. included in the system. The present invention can also be implemented by computer software that executes the methods of the present invention, for example, by control software executed by a microprocessor, electronic control unit, client, server, etc. However, it should be noted that the computer software executing the methods of the present invention is not limited to execution in one or a specific set of hardware entities; it can also be implemented in a distributed manner by unspecified hardware. For computer software, the software product can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.) or distributed across a network, as long as it enables electronic devices to execute the methods according to the present invention.

[0157] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described herein. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the specific embodiments described above. Therefore, any modifications or equivalent substitutions to the present invention, as well as all technical solutions and improvements that do not depart from the spirit and scope of the invention, are covered within the scope of the claims of the present invention.

Claims

1. A log retrieval method based on distributed storage, characterized in that, include: The logs are categorized according to custom classification criteria and then split into log blocks corresponding to each category based on the set time dimension. Storing and creating log block indexes for each log block; Receive target log retrieval information, retrieve the target log block corresponding to the target log through the log block index information, and perform a full-text search on the target log block to obtain the retrieval results information of the target log; The process of classifying logs according to custom category criteria and splitting them into corresponding log blocks based on a set time dimension includes: Custom criteria for classifying logs into the same category include: whether they are logs generated by the same machine, and / or whether they belong to the same module, and / or whether they are the same type of logs, or whether they are at the same log level; When dividing logs into log blocks, you can define a single dividing condition or combine multiple dividing conditions to divide logs into multi-level log blocks. Set a uniform time dimension for all types of log blocks, cache all collected logs to the set uniform time dimension, and then split them into log blocks of the corresponding categories; Deploy log collection agents on each machine. The log collection agents obtain log information from the deployed machines and attach the machine name tag, and / or module name tag, and / or log type tag, and / or log level tag when uploading the log information. The process of creating a log block index for each log block includes: An index is created for each log block, and the index content includes: time, log block metadata, and log block storage location; wherein, the log block metadata includes the machine information and / or module information and / or type information and / or level information to which the log block belongs.

2. The log retrieval method based on distributed storage according to claim 1, characterized in that, The module name includes the names of the various services deployed on the machine.

3. A log retrieval method based on distributed storage according to claim 1, characterized in that, The log types include business access logs access.log, and / or database access logs sql.log, and / or business logs server.log; the log types also support business customization, and the log collection agent will attach the log type to the log information for selection of the corresponding log type during retrieval.

4. The log retrieval method based on distributed storage according to claim 1, characterized in that, The log levels mentioned include: INFO indicates that the message highlights the application's execution process at a coarse-grained level; A WARN message indicates a potential error. An ERROR indicates that although an error event has occurred, it does not affect the continued operation of the system.

5. The log retrieval method based on distributed storage according to claim 1, characterized in that, The process of storing each log block and creating a log block index includes: Each log block is stored in multiple levels according to the system's preset time conditions; Log blocks that meet the first preset time condition are stored on the local disk, log blocks that meet the second preset time condition are compressed and uploaded to the remote storage, and cold log block data that meet the third preset time condition are transferred from the remote storage to the archive storage for storage.

6. The log retrieval method based on distributed storage according to claim 1, characterized in that, The process of storing each log block and creating a log block index includes: Multi-level storage is performed on each log block according to the system's preset storage capacity. Log blocks smaller than the first preset storage limit are stored on the local disk. When the number of log blocks stored on the local disk exceeds the first preset storage limit, the log blocks stored on the local disk for a longer period than the first set value are compressed and uploaded to the remote storage for storage. When the number of log blocks stored on the remote storage exceeds the second preset storage limit, cold log block data that has been stored on the remote storage for a longer period than the second set value will be transferred from the remote storage to the archive storage for storage.

7. The log retrieval method based on distributed storage according to claim 1, characterized in that, When performing a target log retrieval, the target log block corresponding to the target log is retrieved through the log block index information, and a full-text search is performed on the target log block to obtain the retrieval results information of the target log, including: When performing target log retrieval, extract key information from the acquired target log that corresponds to the log block classification category; The key information obtained is used to retrieve the target log block and its storage location from the log block index information; Download the target log blocks stored on remote storage or archive storage to the local disk, and then perform a full-text search on each log block using the search information of the target log as the search criteria. The search results are then summarized and sorted to obtain the search results information.

8. A log retrieval device based on distributed storage, characterized in that, include: The log block classification module categorizes logs according to custom classification criteria and splits them into log blocks corresponding to each category based on a set time dimension. The log block storage module stores individual log blocks. And a log block retrieval module, which establishes a log block index for the stored log blocks. The log block retrieval module receives the target log retrieval information, retrieves the target log block corresponding to the target log through the log block index information, and performs a full-text search on the target log block to obtain the retrieval result information of the target log. The process of classifying logs according to custom category criteria and splitting them into corresponding log blocks based on a set time dimension includes: Custom criteria for classifying logs into the same category include: whether they are logs generated by the same machine, and / or whether they belong to the same module, and / or whether they are the same type of logs, or whether they are at the same log level; When dividing logs into log blocks, you can define a single dividing condition or combine multiple dividing conditions to divide logs into multi-level log blocks. Set a uniform time dimension for all types of log blocks, cache all collected logs to the set uniform time dimension, and then split them into log blocks of the corresponding categories; Deploy log collection agents on each machine. The log collection agents obtain log information from the deployed machines and attach the machine name tag, and / or module name tag, and / or log type tag, and / or log level tag when uploading the log information. The process of creating a log block index for each log block includes: An index is created for each log block, and the index content includes: time, log block metadata, and log block storage location; wherein, the log block metadata includes the machine information and / or module information and / or type information and / or level information to which the log block belongs.

9. A storage medium, characterized in that, The system contains a computer-executable program, which, when executed, implements a massive log retrieval method based on distributed storage as described in any one of claims 1-7.

Images