Log processing method and apparatus, and electronic device

Abstract

The present disclosure relates to a log processing method and device and electronic equipment, and relates to the technical field of data processing. The method comprises: receiving target log label information and an execution statement input by a user, the execution statement comprising a first execution statement and a second execution statement; determining a log block indicated by the target log label information in an index list, and performing concurrent query processing on multiple log blocks by using the first execution statement to obtain multiple query results of the multiple log blocks, the log block being a log set in a same time range, and different log blocks corresponding to different log indexes in the index list; and performing aggregation processing on the multiple query results by using the second execution statement, and sending an aggregation result to the user. Through the scheme of the present disclosure, the workload of log processing can be reduced, and the consumption of a central processing unit and storage space can be reduced.

Description

Log processing methods, devices and electronic equipment Technical Field

[0001] This application relates to the field of data processing technology, specifically to a log processing method, apparatus, and electronic device. Background Technology

[0002] As applications become increasingly distributed and microservice-based, service traffic in systems rises, leading to a surge in overall log volume. In such cases, a single machine struggles to store all logs. Distributed open-source search engines (Elasticsearch) or columnar database management systems (ClickHouse) are typically used to store large numbers of individual log entries. Storing these numerous individual log entries requires parsing each one individually into key-value pairs. Since an index needs to be created for each log entry, and log queries require a lookup of each index, the workload of log processing increases significantly. Furthermore, the log parsing and indexing process consumes more CPU and storage space. Summary of the Invention

[0003] In view of this, the present disclosure provides a log processing method, apparatus and electronic device, the main purpose of which is to solve the technical problems of large workload in current log processing and increased consumption of central processing unit and storage space due to log parsing and indexing processes.

[0004] According to a first aspect of this disclosure, a log processing method is provided, the method comprising:

[0005] Receive target log tag information and execution statements input by the user, wherein the execution statements include a first execution statement and a second execution statement;

[0006] The target log tag information indicates the log block to be queried in the index list, and the first execution statement is used to perform concurrent query processing on multiple log blocks to obtain multiple query results for multiple log blocks. The log blocks are a collection of logs within the same time range, and different log blocks correspond to different log indexes in the index list.

[0007] The second execution statement is used to aggregate the multiple query results, and the aggregated results are sent to the user.

[0008] According to a second aspect of this disclosure, a log processing apparatus is provided, the apparatus comprising:

[0009] The receiving module is used to receive target log tag information and execution statements input by the user, wherein the execution statements include a first execution statement and a second execution statement;

[0010] The query module is used to determine the log block indicated by the target log tag information in the index list, and to perform concurrent query processing on multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks. The log block is a collection of logs within the same time range, and different log blocks correspond to different log indexes in the index list.

[0011] The aggregation module is used to aggregate the multiple query results using the second execution statement and send the aggregated results to the user.

[0012] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect described above.

[0013] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method of the first aspect described above.

[0014] Compared with the prior art, the log processing method, apparatus, and electronic device provided in this disclosure receive target log tag information and execution statements input by the user. The execution statements include a first execution statement and a second execution statement. The method determines the log block indicated by the target log tag information in an index list, and performs concurrent query processing on multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks. Each log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list. The second execution statement aggregates the multiple query results and sends the aggregated results to the user. By applying the scheme of this disclosure, the scope of log searching can be determined by using log tag information to find the corresponding log block in the index list. By performing concurrent query processing on the log block, the workload of log processing is reduced. Since the log blocks in the index list are log collections, creating corresponding log indexes for the log collections can reduce the consumption of central processing unit and storage space. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a schematic flowchart of a log processing method provided in an embodiment of this disclosure;

[0018] Figure 2 is a schematic flowchart of a log processing method provided in another embodiment of this disclosure;

[0019] Figure 3 is a schematic diagram of the structure of a log processing device provided in an embodiment of this disclosure;

[0020] Figure 4 is a schematic diagram of the structure of a log processing device provided in an embodiment of this disclosure. Detailed Implementation

[0021] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0022] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0023] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] The log processing method, apparatus, and electronic device of this disclosure are described below with reference to the accompanying drawings.

[0026] This disclosure provides a log processing method, apparatus, and electronic device that can reduce the workload of log processing and reduce the consumption of central processing unit and storage space when processing logs.

[0027] As shown in Figure 1, an embodiment of this disclosure provides a log processing method, including:

[0028] Step 101: Receive the target log tag information and execution statement input by the user. The execution statement includes a first execution statement and a second execution statement.

[0029] The target log is the specific log that the user wants to query. The target log tag information is one or more feature dimensions corresponding to the target log that the user wants to query. The feature dimensions may include, but are not limited to, cluster name, service name, instance name, target log type, target log time range, etc. The target log time range may include the target log start time, target log end time, etc. The first execution statement is the command for the specific target log that the user wants to query, such as querying the number of log lines containing "test-string" in the target log, where "test-string" is the file name of the target log being searched. The second execution statement is the command for aggregating the specific target log, such as summing the number of log lines containing "test-string" in the target log.

[0030] In this embodiment of the disclosure, the executing entity may be a log processing device, which may include, but is not limited to, an indexing module, a querying module, an aggregation module, an analysis module, etc., without specific limitations. The log processing device can determine the corresponding search range of the target log by receiving the target log tag information, and query the specific log information of the corresponding target log within the search range by executing statements, thereby reducing the workload of log processing. For a detailed description of the implementation process, please refer to step 102 of the embodiment.

[0031] Step 102: Determine the log block to be queried in the index list by the target log tag information, and use the first execution statement to perform concurrent query processing on multiple log blocks to obtain multiple query results for multiple log blocks.

[0032] The index list is a pre-created list of target logs used for locating logs by the index module. The index list includes log blocks and their corresponding log indexes. A log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list. The log index contains the log tag information corresponding to the log block. The log tag information may include the cluster name, service name, instance name, log block type, and log block time range. The log block time range may include the log block start time and log block end time.

[0033] In one possible implementation of this disclosure, the target log tag information can be matched with the log tag information corresponding to each log index in the index list; the log block whose corresponding log tag information successfully matches the target log tag information is determined as the log block queried by the target log tag information.

[0034] In a specific application scenario, for example, a user inputs the target log tag information they want to query: the target log time range is from 9:00 AM to 9:30 AM, and the target log type is system log. The log blocks in the index list whose corresponding log index information has a log block time range of 9:00 AM to 9:30 AM and a log block type of system log are identified as the log blocks indicated by the target log tag information. By identifying the log blocks corresponding to the target log tag information, log blocks that do not match the target log can be filtered out, thereby narrowing the search scope of the target log and reducing the workload of log processing.

[0035] In this embodiment of the disclosure, concurrent query processing involves using a query module to concurrently query multiple log blocks for specific target logs according to a first execution statement. This concurrent query processing may include, but is not limited to, map operations. For example, the first execution statement may be to query the number of log lines containing the string "test-string" in the target log. The concurrent query processing executes the command of the first execution statement individually on each log block to obtain the number of log lines containing "test-string" in that log block. After the query is completed, the obtained query results include at least one of the target logs found in the log block and the data query status of each data block. The data query status includes a normal status and an abnormal status. A normal status indicates a normal data query status when the first execution statement performs concurrent query processing on the log block, i.e., the target log is successfully found in the log block. An abnormal status indicates an abnormal data query status when the first execution statement performs concurrent query processing on the log block, i.e., the target log is not found in the log block.

[0036] Step 103: Use the second execution statement to aggregate multiple query results and send the aggregated results to the user.

[0037] The aggregation process involves using the aggregation module to summarize multiple query results output by the concurrent query processing according to the second execution statement. The aggregation process may include, but is not limited to, reduction function operations.

[0038] For an example embodiment of this disclosure, the second execution statement is to sum the number of log lines containing "test-string" in the target log. The aggregation process will sum the number of log lines containing "test-string" once and send the summation result to the user.

[0039] In summary, according to the log processing method provided in this disclosure, compared with the prior art, this disclosure receives target log tag information and execution statements input by the user. The execution statements include a first execution statement and a second execution statement. The target log tag information indicates the log block to be queried in the index list. The first execution statement is used to perform concurrent query processing on multiple log blocks, resulting in multiple query results for multiple log blocks. Each log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list. The second execution statement is used to aggregate the multiple query results, and the aggregated results are sent to the user. By applying the scheme of this disclosure, the scope of log searching can be determined by using log tag information to find the corresponding log block in the index list. By performing concurrent query processing on the log block, the workload of log processing is reduced. Since the log blocks in the index list are log collections, creating corresponding log indexes for the log collections can reduce the consumption of central processing unit and storage space.

[0040] Furthermore, as a refinement and extension of the above embodiments, and to fully illustrate the specific implementation process of the method in this embodiment, this embodiment provides a specific method as shown in Figure 2, which includes:

[0041] Step 201: Obtain multiple logs of the same dimension, store the multiple logs into log blocks corresponding to different time ranges according to time range, and create an index list. The index list includes log blocks and log indexes corresponding to log blocks. Log indexes are used to indicate the log tag information of log blocks.

[0042] Within this category, "same dimension" refers to multiple logs with the same log tag information, such as logs with the same cluster name, service name, instance name, or log type. The time range includes the log's start time and end time.

[0043] In this embodiment of the disclosure, multiple logs corresponding to the same time range within the same dimension can be stored in the same log block, ensuring that the size of each log block is within a preset storage range. This preset storage range may include, but is not limited to, 100MB, to ensure the timeliness of log block processing and improve user experience. Distributed storage of log blocks can include, but is not limited to, storing log blocks on different machines, meeting the needs of storing large numbers of log blocks. It supports on-demand horizontal scaling and assigns appropriate log tag information to each log block, such as the log block's time range, the service to which the log block belongs, the log type, the machine and path where the log block resides, etc. An index list is created using log tag information, with different log blocks corresponding to different log indexes. When querying a target log, the user can specify the corresponding target log tag information. The index module can find the machine where the log block is located according to the path of the log block in the index list, search for the target log on the machine, and return the query results. A log block is a collection of logs within the same time range. The purpose of dividing multiple logs into log blocks is to distinguish target logs in different time periods. If you want to unlock target logs within a certain time period, you only need to find the log blocks in the corresponding time period for processing, which narrows the search range of target logs and reduces the workload of log processing.

[0044] Step 202: Receive user input execution statements, perform syntax analysis on the execution statements, and obtain the first execution statement and the second execution statement.

[0045] The first execution statement is the task instruction to be executed when querying multiple log blocks in parallel, and the second execution statement is the task instruction to be executed when aggregating query results; the execution statements may include, but are not limited to, execution commands in the form of command interpretation (shell) scripts.

[0046] In this embodiment of the disclosure, in order to more accurately find the target log required by the user, the analysis module can be used to perform syntax analysis on the execution statement to obtain the first execution statement and the second execution statement. There are various methods for syntax analysis. One possible approach is to use Term Frequency-Inverse Document Frequency (TF-IDF) technology to extract key words from the executed statements. These key words are then matched with a first preset key word from the concurrent query processing stage. If the first similarity score is greater than a first preset similarity threshold, the first executed statement to be executed in the query stage is obtained. The first preset key word can be a keyword corresponding to the first executed statement pre-set according to actual conditions. The first preset similarity threshold is a value that can be set according to the actual application scenario. The closer the first preset similarity threshold is to 1, the higher the similarity between the key word and the first preset key word from the concurrent query processing stage. Next, the key words are matched with a second preset key word from the aggregation processing stage. If the second similarity score is greater than a second preset similarity threshold, the second executed statement to be executed in the aggregation stage is obtained. The second preset key word can be a keyword corresponding to the second executed statement pre-set according to actual conditions. The second preset similarity threshold is a value that can be set according to the actual application scenario. The closer the second preset similarity threshold is to 1, the higher the similarity between the key word and the second preset key word from the aggregation processing stage.

[0047] In a specific application scenario, for example, the execution statement is `grep test-string|wc-l`. `grep` is a string search command used to find target logs within a log block that match certain conditions. `test-string` is the filename of the target log file being searched, and `wc-l` is a statistics command used to count the number of lines in the target log file named `test-string`. The keywords `grep` and `wc-l` are extracted from the execution statement. `grep` is matched against a first preset keyword for similarity. If the similarity is greater than 90%, the `grep` command and related information are analyzed to obtain the `grep test-string` command to be executed during the query phase. Similarly, `wc-l` is matched against a second preset keyword for similarity. If the similarity is greater than 90%, the `wc-l` command and related information are analyzed to obtain the `wc-l` command to be executed during the aggregation phase.

[0048] Step 203: Receive the target log tag information input by the user, and determine the log block to be queried in the index list according to the target log tag information.

[0049] For the specific implementation process of the embodiments disclosed herein, please refer to the relevant descriptions in steps 101 and 102 of the embodiments, which will not be repeated here.

[0050] Step 204: Use the first execution statement to perform concurrent query processing on multiple log blocks to obtain multiple query results for multiple log blocks.

[0051] In this embodiment of the disclosure, to improve the efficiency of target log querying, a first execution statement can be used to perform concurrent query processing on multiple log blocks to obtain multiple query results for multiple log blocks. The specific implementation process can be described as follows: the first execution statement is used to perform target log query operations on multiple log blocks simultaneously to obtain the first query result of each log block. The first query result includes the target log queried in the log block and at least one of the data query status of each data block; and for log blocks whose data query status is abnormal, the first execution statement is used to repeatedly perform the target log query operation on the log block until the target log is queried, or the number of repetitions is equal to the preset number of repetitions.

[0052] In this embodiment of the disclosure, after obtaining multiple log blocks to be retrieved, a first execution statement is used to simultaneously perform query operations on the target logs to be retrieved on the machines where the log blocks reside, obtaining the retrieved target logs. If the first execution statement successfully retrieves the target logs on the machines where the log blocks reside, the data query status corresponding to the log blocks is normal. If the first execution statement does not retrieve the target logs on the machines where the log blocks reside, the first execution statement is used to repeatedly perform the query operation on the log blocks until the target logs are retrieved, or the number of repetitions equals a preset number of repetitions. The preset number of repetitions is a value that can be set according to the actual application scenario, such as a preset number of repetitions of 3 or 5 times. Finally, the query results after concurrent query processing are stored in the database for easy querying and aggregation of target logs.

[0053] Step 205: Use the second execution statement to aggregate the query results and send the aggregated results to the user.

[0054] In this embodiment of the disclosure, to facilitate users' clear and intuitive understanding of the target log and related information, a second execution statement can be used to aggregate the query results and send the aggregated results to the user. The specific implementation process can be described as follows: determine whether the query results meet the aggregation conditions; if the query results meet the aggregation conditions, use the second execution statement to count the total number of target logs found in multiple log blocks and the total number of multiple log blocks, and perform data aggregation processing on the target logs, and send the target logs, the total number of multiple log blocks, and the aggregated target logs to the user; wherein, the aggregation conditions include at least one of the following conditions: the query duration of the concurrent query processing stage is greater than or equal to a preset duration; or, the concurrent query processing of multiple log blocks is completed.

[0055] In this embodiment of the disclosure, before performing the aggregation processing operation, it is first necessary to determine whether the concurrent query processing operation has been completed. If the concurrent query processing operation is completed, or if the concurrent query processing operation takes too long or the operation is unresponsive for a duration greater than or equal to a preset duration, the aggregation module begins to use the second execution statement to read and count the total number of all log blocks and the total number of log blocks corresponding to the successful query of the target log, as well as to read and count the total number of target logs found in multiple log blocks from the database and perform data aggregation processing on the found target logs. Data aggregation processing may include, but is not limited to, deduplication, summation, and other processing operations. The aggregated target logs, the total number of all log blocks, and the total number of log blocks corresponding to the successful query of the target log are then sent to the user. The user can determine whether the confidence level of the target log retrieval results meets the user's needs based on the proportion of the total number of log blocks corresponding to the successful query of the target log.

[0056] In this embodiment of the disclosure, the problem of how to retrieve aggregated logs on distributed storage is solved by designing a distributed computing (Map Reduce) retrieval scheme. At the same time, by performing concurrent query processing on multiple log blocks on multiple machines at the same time, the efficiency and speed of overall log retrieval and aggregation are significantly improved.

[0057] In summary, according to the log processing method provided in this disclosure, compared with the prior art, this application can receive target log tag information and execution statements input by the user. The execution statements include a first execution statement and a second execution statement. The target log tag information indicates the log block to be queried in the index list. The first execution statement is used to perform concurrent query processing on multiple log blocks, resulting in multiple query results for multiple log blocks. Each log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list. The second execution statement is used to aggregate the multiple query results, and the aggregated results are sent to the user. By applying the scheme of this disclosure, the scope of log searching can be determined by using log tag information to find the corresponding log block in the index list. By performing concurrent query processing on the log block, the workload of log processing is reduced. Since the log blocks in the index list are log collections, creating corresponding log indexes for the log collections can reduce the consumption of central processing unit and storage space.

[0058] Based on the specific implementation of the methods shown in Figures 1 and 2 above, this embodiment provides a log processing device, as shown in Figure 3. The device includes: a receiving module 31, a query module 32, and an aggregation module 33.

[0059] The receiving module 31 is used to receive target log tag information and execution statements input by the user. The execution statements include a first execution statement and a second execution statement.

[0060] The query module 32 is used to determine the log block indicated by the target log tag information in the index list, and to perform concurrent query processing on multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks. The log block is a collection of logs within the same time range, and different log blocks correspond to different log indexes in the index list.

[0061] The aggregation module 33 is used to aggregate multiple query results using the second execution statement and send the aggregated results to the user.

[0062] In specific application scenarios, as shown in Figure 4, the device also includes: a disassembly module 34 and a creation module 35;

[0063] The disassembly module 34 is used to obtain multiple logs of the same dimension and store the multiple logs into log blocks corresponding to different time ranges according to the time range.

[0064] Module 35 is used to create an index list, which includes log blocks and their corresponding log indexes. The log indexes are used to indicate the log tag information of the log blocks.

[0065] In specific application scenarios, the query module 32 can be used to match the target log tag information with the log tag information corresponding to each log index in the index list; the log blocks whose corresponding log tag information successfully matches the target log tag information are identified as the log blocks indicated by the target log tag information for querying.

[0066] In specific application scenarios, as shown in Figure 4, the device also includes: an analysis module 36;

[0067] Analysis module 36 is used to perform syntax analysis on the execution statement to obtain a first execution statement and a second execution statement. The first execution statement is the task instruction to be executed when querying multiple log blocks in parallel, and the second execution statement is the task instruction to be executed when aggregating query results.

[0068] In a specific application scenario, the query module 32 is used to perform a query operation on the target logs of multiple log blocks simultaneously using the first execution statement to obtain the first query result of each log block. The first query result includes the target log found in the log block and at least one of the data query status of each data block; and for log blocks whose data query status is abnormal, the first execution statement is used to repeatedly perform the query operation on the target logs of the log blocks until the target logs are found, or the number of repetitions is equal to the preset number of repetitions.

[0069] In specific application scenarios, the analysis module 36 is used to extract statement keywords from the execution statement, perform similarity matching between the statement keywords and the first preset keywords in the concurrent query processing stage, and if the first similarity is greater than the first preset similarity threshold, then the analysis of the statement keywords yields the first execution statement to be executed in the query stage, and the first preset keyword is the keyword corresponding to the first execution statement; and perform similarity matching between the statement keywords and the second preset keywords in the aggregation processing stage, and if the second similarity is greater than the second preset similarity threshold, then the analysis of the statement keywords yields the second execution statement to be executed in the aggregation stage, and the second preset keyword is the keyword corresponding to the second execution statement.

[0070] In a specific application scenario, the query module 32 is used to determine whether the query result meets the aggregation condition. If the query result meets the aggregation condition, the second execution statement is used to count the total number of target logs and the total number of multiple log blocks found in multiple log blocks, and to perform data aggregation processing on the target logs. The target logs, the total number of multiple log blocks, and the aggregated target logs are then sent to the user. The aggregation condition includes at least one of the following conditions: the query duration of the concurrent query processing phase is greater than or equal to the preset duration; or, concurrent query processing of multiple log blocks is completed.

[0071] Based on the methods shown in Figures 1 and 2, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the methods shown in Figures 1 and 2.

[0072] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0073] Based on the methods shown in Figures 1 and 2, and the virtual device embodiments shown in Figures 3 and 4, in order to achieve the above objectives, this application also provides an electronic device that can be configured on the end side of a vehicle (such as an electric vehicle). The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the methods shown in Figures 1 and 2.

[0074] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0075] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0076] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms, or it can be implemented by hardware. Compared with the prior art, the technical solution of this disclosure receives target log tag information and execution statements input by the user. The execution statements include a first execution statement and a second execution statement. The target log tag information indicates the log block to be queried in the index list, and the first execution statement is used to perform concurrent query processing on multiple log blocks to obtain multiple query results for multiple log blocks. The log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list. The second execution statement is used to aggregate the multiple query results and send the aggregated results to the user. By applying the solution of this disclosure, the scope of log search can be determined by using log tag information to find the corresponding log block in the index list. By performing concurrent query processing on the log block, the workload of log processing is reduced. Since the log blocks in the index list are log collections, creating corresponding log indexes for the log collections can reduce the consumption of central processing unit and storage space.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0079] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A log processing method, characterized in that, The method includes: receiving target log tag information and execution statements input by a user, the execution statements including a first execution statement and a second execution statement; determining the log block indicated by the target log tag information in an index list, and performing concurrent query processing on multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks, wherein the log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list; the index list is a list pre-created by an index module for finding target logs; aggregating the multiple query results using the second execution statement, and sending the aggregated results to the user; wherein the method further includes: obtaining multiple logs of the same dimension, storing the multiple logs into log blocks corresponding to different time ranges according to their time ranges; creating an index list, the index list including the log blocks and the log indexes corresponding to the log blocks, the log indexes being used to indicate the log tags of the log blocks. Information; wherein, determining the log block indicated by the target log tag information in the index list includes: matching the target log tag information with the log tag information corresponding to each log index in the index list; determining the log block whose corresponding log tag information successfully matches the target log tag information as the log block indicated by the target log tag information; concurrently querying multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks includes: simultaneously performing a target log query operation on multiple log blocks using the first execution statement to obtain the first query result for each log block, the first query result including at least one of the target log found in the log block and the data query status of each data block; and, for log blocks whose data query status is abnormal, repeatedly performing the target log query operation on the log block using the first execution statement until the target log is found, or the number of repetitions equals a preset number of repetitions.

2. The method according to claim 1, characterized in that, The method further includes: performing syntax analysis on the execution statement to obtain a first execution statement and a second execution statement, wherein the first execution statement is a task instruction to be executed when querying multiple log blocks in parallel, and the second execution statement is a task instruction to be executed when aggregating the query results.

3. The method according to claim 2, characterized in that, The step of performing syntactic analysis on the executed statement to obtain a first executed statement and a second executed statement includes: extracting statement keywords from the executed statement; performing similarity matching between the statement keywords and a first preset keyword in the concurrent query processing stage; if the first similarity is greater than a first preset similarity threshold, then analyzing the statement keywords to obtain a first executed statement to be executed in the query stage, wherein the first preset keyword is the keyword corresponding to the first executed statement; performing similarity matching between the statement keywords and a second preset keyword in the aggregation processing stage; if the second similarity is greater than a second preset similarity threshold, then analyzing the statement keywords to obtain a second executed statement to be executed in the aggregation stage, wherein the second preset keyword is the keyword corresponding to the second executed statement.

4. The method according to claim 1, characterized in that, The step of using the second execution statement to aggregate the query results and sending the aggregated results to the user includes: determining whether the query results meet the aggregation conditions; if the query results meet the aggregation conditions, then using the second execution statement to count the total number of the target logs found in the multiple log blocks and the total number of the multiple log blocks, and performing data aggregation processing on the target logs, and sending the target logs, the total number of the multiple log blocks, and the aggregated target logs to the user; wherein, the aggregation conditions include at least one of the following conditions: the query duration of the concurrent query processing phase is greater than or equal to a preset duration; or, completing the concurrent query processing of multiple log blocks.

5. A log processing device, characterized in that, include: The receiving module is used to receive target log tag information and execution statements input by the user, wherein the execution statements include a first execution statement and a second execution statement; The query module is used to determine the log block indicated by the target log tag information in the index list, and to perform concurrent query processing on multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks. The log block is a collection of logs within the same time range, and different log blocks correspond to different log indices in the index list. The index list is a list pre-created by the index module for finding target logs. An aggregation module is used to aggregate the multiple query results using the second execution statement and send the aggregated results to the user; a decomposition module is used to obtain multiple logs of the same dimension and store the multiple logs into log blocks corresponding to different time ranges according to their time ranges; a creation module is used to create an index list, which includes log blocks and log indexes corresponding to the log blocks, and the log indexes are used to indicate the log tag information of the log blocks; wherein, determining the log block to be queried by the target log tag information in the index list includes: matching the target log tag information with the log tag information corresponding to each log index in the index list; matching the corresponding log tag information with the target log tag information. A log block that successfully matches the information is identified as the log block indicated by the target log tag information for querying; the step of concurrently querying multiple log blocks using the first execution statement to obtain multiple query results for multiple log blocks includes: simultaneously performing a target log query operation on multiple log blocks using the first execution statement to obtain an initial query result for each log block, the initial query result including at least one of the target log found in the log block and the data query status of each data block; and, for log blocks whose data query status is abnormal, repeatedly performing the target log query operation on the log block using the first execution statement until the target log is found, or the number of repetitions equals a preset number of repetitions.

6. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images