Log statistical query method and device, equipment and medium

Abstract

The invention discloses a log statistical query method and device, equipment and a medium. The method comprises the following steps: determining a query mode of each log statistical query request, marking the query mode meeting a high-frequency condition as a high-frequency query mode, and regularly obtaining business logs meeting the high-frequency query mode from a business log library to obtain accelerated aggregation data; when at least one query mode is marked as a high-frequency query mode, after a log statistics query request is obtained, determining the query mode of the log statistics query request, and detecting whether a matched target high-frequency query mode exists or not; and if yes, determining a query result according to the target time interval, the accelerated aggregation data and the business logs in the business log library. According to the embodiment of the invention, the query result can be determined based on the pre-acquired business log and the original business log in the business log library, so that the number of the business logs required to be acquired from the business log library in real time is remarkably reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a log statistics query method, apparatus, device, and medium. Background Technology

[0002] During each business processing operation, an enterprise's business system generates business logs. These logs record data related to the business processes. They contain the time information of the operation and field information for each business parameter. The parameter fields include the parameter name and value. Enterprises typically have data storage devices containing a business log repository. The business system stores its generated business logs in this repository. During system operation, business personnel may send log statistics query requests to the data storage device. These requests request statistical analysis of the parameter values ​​of a target business parameter within a target time interval in the business logs. The system then performs a statistical query based on these requests to obtain the query results.

[0003] In related technologies, a common log statistics query scheme involves retrieving relevant business logs from a business log database in real time after each log statistics query request is received. Statistics are then performed on these retrieved business logs to obtain the query results. However, this approach requires retrieving relevant business logs from a large business log database in real time after each request, resulting in long query times and significant resource consumption during the query process. Summary of the Invention

[0004] This invention provides a log statistics query method, apparatus, device, and medium to solve the problems of long query time and high resource consumption in related technologies.

[0005] According to one aspect of the present invention, a log statistics query method is provided, comprising: After obtaining the query instruction information, the system begins to process each log statistics query request, determines the query mode of each log statistics query request, continuously counts the number of times the determined query mode is triggered within a preset time window, marks the query modes that meet the high-frequency conditions as high-frequency query modes, and periodically retrieves business logs that meet the high-frequency query modes from the business log library to obtain accelerated aggregation data of high-frequency query modes. When at least one query pattern is marked as a high-frequency query pattern, after obtaining the log statistics query request, determine the query pattern of the log statistics query request and check whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request. If a target high-frequency query pattern exists, the query result of the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregated data of the target high-frequency query pattern, and the business logs in the business log library, and the query result is output.

[0006] According to another aspect of the present invention, a log statistics query device is provided, comprising: The pattern processing module is used to process each log statistics query request after obtaining the query instruction information, determine the query pattern of each log statistics query request, continuously count the number of times the determined query pattern is triggered within a preset time window, mark the query patterns that meet the high-frequency conditions as high-frequency query patterns, and periodically obtain business logs that meet the high-frequency query patterns from the business log library to obtain accelerated aggregation data of high-frequency query patterns. The pattern matching module is used to determine the query pattern of the log statistics query request after obtaining the log statistics query request when at least one query pattern is marked as a high-frequency query pattern, and to detect whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request. The pattern query module is used to determine the query result of the log statistics query request based on the target time interval of the log statistics query request, the accelerated aggregated data of the target high-frequency query pattern, and the business logs in the business log library, and output the query result if a target high-frequency query pattern exists.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to execute the log statistics query method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the log statistics query method described in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the log statistics query method described in any embodiment of the present invention.

[0010] The technical solution of this invention involves processing each log statistics query request after obtaining query instruction information, determining the query mode of each log statistics query request, continuously counting the number of times the determined query mode is triggered within a preset time window, marking query modes that meet high-frequency conditions as high-frequency query modes, and periodically retrieving business logs that match the high-frequency query modes from the business log library to obtain accelerated aggregation data for the high-frequency query modes. When at least one query mode is marked as a high-frequency query mode, after obtaining a log statistics query request, the query mode of the log statistics query request is determined, and it is detected whether there is a target high-frequency query mode that matches the query mode of the log statistics query request. If a target high-frequency query mode exists, the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query mode, and the business logs in the business log library. This solution addresses the problem in related log statistics query schemes where, after each log statistics query request, the relevant business logs need to be retrieved from a business log database containing a large number of business logs in real time, resulting in long query times and high resource consumption. It automatically identifies and statistically analyzes the query patterns of log statistics query requests. For frequently occurring query patterns, it pre-retrieves business logs matching the query pattern from the business log database. Then, when subsequent log statistics query requests with frequently occurring patterns are received, the query results are determined based on the pre-retrieved business logs and the original business logs in the business log database. This significantly reduces the number of business logs that need to be retrieved from the business log database in real time, reduces query time, lowers resource consumption during the query process, and improves query response speed.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a log statistics query method provided in Embodiment 1 of the present invention.

[0014] Figure 2 This is a flowchart of a log statistics query method provided in Embodiment 2 of the present invention.

[0015] Figure 3 This is a schematic diagram of the structure of a log statistics query device provided in Embodiment 3 of the present invention.

[0016] Figure 4 A schematic diagram of the structure of an electronic device for implementing the log statistics query method of this invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] It should be noted that the terms "target," "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising," "including," and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Example 1 Figure 1 This is a flowchart illustrating a log statistics query method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where a statistical query is performed based on a log statistics query request to obtain the query results. This method can be executed by a log statistics query device, which can be implemented in hardware and / or software and can be configured in a data storage device within an enterprise. The data storage device is an electronic device used to store business logs. The data storage device contains a business log database. The business log database is a database used to store business logs. For example... Figure 1 As shown, the method includes: Step 101: After obtaining the query instruction information, start processing each log statistics query request, determine the query mode of each log statistics query request, continuously count the number of times the determined query mode is triggered within the preset time window, mark the query mode that meets the high-frequency condition as the high-frequency query mode, and periodically obtain the business logs that meet the high-frequency query mode from the business log library to obtain the accelerated aggregation data of the high-frequency query mode.

[0020] Optionally, the enterprise's business system can be a server used to manage the enterprise's business. During each business processing operation, the business system generates a business log. The business system stores these logs in a business log repository. A business processing operation can be an operation performed by the business system that is related to the enterprise's business. The enterprise's business can be producing software products with specified functions, producing hardware products with specified functions, providing specified services to specified individuals, or providing specified services to specified other enterprises. The business log is used to record data related to business processing operations. The business log contains time information for the business processing operation and field information for each business parameter of the operation. The time information for the business processing operation can be the time recorded by the business system when it began executing the business processing operation. Each business parameter can be multiple parameters that can be used to describe the business processing operation. The field information for each business parameter includes the parameter name and parameter value. The parameter name for each business parameter can be a pre-set text used to uniquely identify the business parameter. The parameter value for each business parameter can be the specific content of the business parameter recorded by the business system.

[0021] Optionally, and exemplaryly, the various business parameters of a business processing operation include, but are not limited to: operation address, operation type, operation duration, and operation result. The operation address can be information describing the Internet Protocol Address (IP) used during the execution of the business processing operation. The operation type can be information describing the type of business processing operation. The operation duration can be information describing the time taken to execute the business processing operation. The operation result can be information describing whether the business processing operation was successfully executed. The parameter value of the operation address can be the IP address recorded by the business system during the execution of the business processing operation. The parameter value of the operation type can be text generated by the business system to describe the type of business processing operation. The parameter value of the operation duration can be the time taken to execute the business processing operation recorded by the business system. The parameter value of the operation result can be text generated by the business system to describe whether the business processing operation was successfully executed.

[0022] Optionally, the business personnel can be technical staff within the enterprise responsible for managing the enterprise's business. During the operation of the business system, the business personnel will send log statistics query requests to the data storage device. A log statistics query request can be information used to request statistical queries on the parameter values ​​of a target business parameter in business logs within a target time interval in the business log database. The log statistics query request includes a target time interval. The target time interval can be a specified time range. Business logs within the target time interval refer to business logs whose time information falls within the target time interval. The target business parameter can be a specified business parameter. Statistical queries on the parameter values ​​of the target business parameter in the business logs within the target time interval can be used to count the total number of business logs within the target time interval whose target business parameter value is the specified parameter value. The log statistics query request includes the parameter name of the target business parameter and the specified parameter value. The query result of the log statistics query request refers to the total number of business logs in the business log database whose time information falls within the target time interval of the log statistics query request and whose parameter name is the same as the parameter name in the log statistics query request, and whose parameter value is the specified parameter value in the log statistics query request. For example, the target time interval is the 24 hours prior to the current time. The target service parameter is the operation address. The specified parameter value is a specific IP address. A statistical query of the target service parameter values ​​in the service logs within a target time interval can be used to count the total number of service logs within a 24-hour period whose operation address parameter value is the specified IP address.

[0023] Optionally, the query instruction information can be information instructing the data storage device to begin statistical queries based on the received log statistics query requests. It can detect whether the data storage device has received the query instruction information. After detecting that the data storage device has received the query instruction information, it can begin detecting whether the data storage device has received the log statistics query request. Each time a log statistics query request is detected, the time when the data storage device receives the request is recorded, and the received log statistics query requests are obtained. Each obtained log statistics query request can be processed, and the query patterns of each request can be monitored. The number of times each query pattern is triggered within a preset time window is continuously counted. Query patterns that meet high-frequency conditions are marked as high-frequency query patterns. Business logs matching the high-frequency query patterns are periodically retrieved from the business log database to obtain accelerated aggregation data for the high-frequency query patterns.

[0024] Optionally, each obtained log statistics query request is processed, including: for each log statistics query request, performing the following processing: retrieving business logs from the business log database that match the target time range, parameter name, and specified parameter value in the log statistics query request; counting the total number of retrieved business logs to obtain the query result of the log statistics query request; and outputting the query result of the log statistics query request. Business logs that match the target time range, parameter name, and specified parameter value in the log statistics query request refer to business logs whose time information is within the target time range in the log statistics query request, whose parameter name is the same as the parameter name in the log statistics query request, and whose parameter value is the specified parameter value in the log statistics query request. Outputting the query result of the log statistics query request can mean sending the query result of the log statistics query request to the terminal device of the user who submitted the query. The user who submitted the query can refer to the business personnel who sent the log statistics query request.

[0025] Optionally, the query pattern of a log statistics query request can refer to the parameter names and specified parameter values ​​included in the log statistics query request. A high-frequency query pattern can refer to a query pattern that frequently appears in log statistics query requests retrieved from the data storage device. Business logs conforming to a high-frequency query pattern can refer to business logs whose parameter names are the same as those in the high-frequency query pattern, and whose parameter values ​​are the specified parameter values ​​in the high-frequency query pattern. Accelerated aggregation data based on the high-frequency query pattern can refer to business logs that conform to the high-frequency query pattern retrieved from the business log database.

[0026] Optionally, the query pattern for each log statistics query request is determined, including: for each obtained log statistics query request, determining the query pattern based on the key query parameters in the log statistics query request. The key query parameters in the log statistics query request can refer to the parameter names and specified parameter values ​​contained in the log statistics query request. The key query parameters can be extracted from the log statistics query request, and these extracted key query parameters can be used to determine the query pattern for the log statistics query request.

[0027] Optionally, continuously count the number of times a determined query pattern is triggered within a preset time window, including: for each determined query pattern, continuously count the number of times the query pattern is triggered within the preset time window. The preset time window can refer to a preset duration prior to the current moment. The preset duration can be a pre-set duration. For each determined query pattern, the number of times the query pattern is triggered within the preset time window can be the total number of log statistics query requests with that query pattern among all log statistics query requests received within the preset time window.

[0028] Optionally, query patterns that meet the high-frequency conditions are marked as high-frequency query patterns, including: determining query patterns that are triggered more than a preset threshold number of times within a preset time window as high-frequency query patterns, and storing the high-frequency query patterns in the high-frequency pattern storage component.

[0029] Optionally, the high-frequency pattern storage component can be a pre-configured software or hardware module for storing high-frequency query patterns. The preset frequency threshold can be a pre-set threshold value. Generally, if the number of times a query pattern is triggered within a preset time window exceeds the preset frequency threshold, the query pattern can be determined to be a high-frequency query pattern.

[0030] Optionally, before storing the high-frequency query pattern in the high-frequency query pattern storage component, it can be checked whether the same high-frequency query pattern has already been stored. If the same high-frequency query pattern has already been stored, then storing the high-frequency query pattern in the high-frequency query pattern storage component will not be performed. If the same high-frequency query pattern has not been stored, then the high-frequency query pattern will be stored in the high-frequency query pattern storage component.

[0031] Optionally, business logs conforming to high-frequency query patterns are periodically retrieved from the business log library to obtain accelerated aggregation data for high-frequency query patterns. This includes performing the following operations for each high-frequency query pattern: retrieving business logs conforming to high-frequency query patterns from the business log library periodically according to a preset time granularity, and storing the retrieved business logs in the data storage component corresponding to the high-frequency query pattern; wherein, the business logs conforming to high-frequency query patterns stored in the data storage component corresponding to the high-frequency query pattern constitute the accelerated aggregation data for high-frequency query patterns.

[0032] Optionally, the data storage component corresponding to the high-frequency query pattern can be a file used to store accelerated aggregation data for the high-frequency query pattern. A blank file can be created and designated as the data storage component corresponding to the high-frequency query pattern.

[0033] Optionally, the preset time granularity can be a pre-set duration. For example, the preset time granularity is 1 hour. According to the preset time granularity, business logs conforming to a high-frequency query pattern are periodically retrieved from the business log library, and the retrieved business logs are stored in the data storage component corresponding to the high-frequency query pattern. This includes: performing a pre-aggregation operation corresponding to the high-frequency query pattern once every preset time granularity. The pre-aggregation operation corresponding to the high-frequency query pattern refers to retrieving business logs from the business log library whose time information is within a preset time granularity prior to the current time and conforms to the high-frequency query pattern, and storing the retrieved business logs in the data storage component corresponding to the high-frequency query pattern.

[0034] Step 102: When at least one query pattern is marked as a high-frequency query pattern, after obtaining the log statistics query request, determine the query pattern of the log statistics query request and detect whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request.

[0035] Optionally, at least one query pattern is marked as a high-frequency query pattern, meaning that the high-frequency pattern storage component stores at least one high-frequency query pattern. After obtaining a log statistics query request, the log statistics query request is no longer processed directly. Instead, the query pattern of the log statistics query request is determined, and it is checked whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request.

[0036] Optionally, the query mode of the log statistics query request can be determined, including: determining the query mode of the log statistics query request based on the key query parameters in the log statistics query request.

[0037] Optionally, the target high-frequency query pattern matching the query pattern of the log statistics query request refers to the high-frequency query pattern stored in the high-frequency pattern storage component that is the same as the query pattern of the log statistics query request. Detecting whether a target high-frequency query pattern matching the query pattern of the log statistics query request exists includes: detecting whether the high-frequency pattern storage component stores a high-frequency query pattern that is the same as the query pattern of the log statistics query request; if the high-frequency pattern storage component stores a high-frequency query pattern that is the same as the query pattern of the log statistics query request, then it is determined that a target high-frequency query pattern matching the query pattern of the log statistics query request exists, and the high-frequency query pattern that is the same as the query pattern of the log statistics query request is determined as the target high-frequency query pattern matching the query pattern of the log statistics query request; if the high-frequency pattern storage component does not store a high-frequency query pattern that is the same as the query pattern of the log statistics query request, then it is determined that no target high-frequency query pattern matching the query pattern of the log statistics query request exists.

[0038] Step 103: If a target high-frequency query pattern exists, determine the query result of the log statistics query request based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query pattern, and the business logs in the business log library, and output the query result.

[0039] Optionally, the query result of the log statistics query request refers to the total number of business logs in the business log library whose time information is within the target time interval in the log statistics query request and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request.

[0040] Optionally, the query result of the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query pattern, and the business logs in the business log library. This includes: if the target time interval of the log statistics query request includes both aggregated and unaggregated intervals, then the query result of the log statistics query request in the aggregated interval is determined based on the accelerated aggregation data of the target high-frequency query pattern; the query result of the log statistics query request in the unaggregated interval is determined based on the business logs in the business log library; and the query results of the log statistics query request in the aggregated and unaggregated intervals are merged to obtain the query result of the log statistics query request.

[0041] Optionally, if a target high-frequency query pattern exists, it can be identified whether the target time interval includes aggregated and unaggregated intervals. An aggregated interval can refer to a sub-time interval within the target time interval where pre-aggregation of business logs matching the target high-frequency query pattern has already been performed. An unaggregated interval can refer to a sub-time interval within the target time interval where pre-aggregation of business logs matching the target high-frequency query pattern has not yet been performed.

[0042] Optionally, the query result of the log statistics query request within the aggregated interval refers to the total number of business logs whose time information is within the aggregated interval and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request. Based on the accelerated aggregation data of the target high-frequency query pattern, determining the query result of the log statistics query request within the aggregated interval includes: obtaining business logs stored in the data storage component corresponding to the high-frequency query pattern whose time information is within the aggregated interval and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request; and counting the total number of obtained business logs to obtain the query result of the log statistics query request within the aggregated interval.

[0043] Optionally, the query result of the log statistics query request in the unaggregated range refers to the total number of business logs whose time information is within the unaggregated range and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request. Determining the query result of the log statistics query request in the unaggregated range based on the business logs in the business log library includes: obtaining business logs stored in the business log library whose time information is within the unaggregated range and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request; and counting the total number of obtained business logs to obtain the query result of the log statistics query request in the unaggregated range.

[0044] Optionally, the query results of the log statistics query request in the aggregated and unaggregated intervals are merged to obtain the query result of the log statistics query request. This includes summing the query results of the log statistics query request in the aggregated interval and the query results of the log statistics query request in the unaggregated interval, and determining the sum as the query result of the log statistics query request. The sum of the query results of the log statistics query request in the aggregated interval and the query results of the log statistics query request in the unaggregated interval is the total number of business logs in the business log database whose time information is within the target time interval in the log statistics query request and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request.

[0045] Optionally, the query result of the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregated data of the target high-frequency query pattern, and the business logs in the business log library. This also includes: if the target time interval of the log statistics query request only includes the aggregated interval, then the query result of the log statistics query request is determined based on the accelerated aggregated data of the target high-frequency query pattern.

[0046] Optionally, if the target time interval of the log statistics query request only includes the aggregated interval, the total number of business logs stored in the data storage component corresponding to the high-frequency query mode, whose time information is within the target time interval of the log statistics query request and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request, is the query result of the log statistics query request. Determining the query result of the log statistics query request based on the accelerated aggregated data of the target high-frequency query mode includes: obtaining business logs stored in the data storage component corresponding to the high-frequency query mode, whose time information is within the target time interval of the log statistics query request and whose parameter names are the same as the parameter names in the log statistics query request, and whose parameter values ​​are the specified parameter values ​​in the log statistics query request; and counting the total number of obtained business logs to obtain the query result of the log statistics query request.

[0047] Optionally, the query result of the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregated data of the target high-frequency query pattern, and the business logs in the business log library. This also includes: if the target time interval of the log statistics query request only contains unaggregated intervals, then business logs that match the target time interval, parameter name, and specified parameter value in the log statistics query request are obtained from the business log library; the total number of obtained business logs is counted to obtain the query result of the log statistics query request.

[0048] Optionally, after detecting whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request, the method further includes: if there is no target high-frequency query pattern, then obtaining business logs from the business log library that match the target time interval, parameter name, and specified parameter value in the log statistics query request; counting the total number of obtained business logs, obtaining the query result of the log statistics query request, and outputting the query result.

[0049] Optionally, it also includes: when no query mode is marked as a high-frequency query mode, after receiving a log statistics query request, retrieving business logs from the business log database that match the target time range, parameter name, and specified parameter value in the log statistics query request, counting the total number of retrieved business logs, obtaining the query result of the log statistics query request, and outputting the query result. No query mode being marked as a high-frequency query mode means that the high-frequency mode storage component does not store high-frequency query modes.

[0050] Optionally, it also includes: periodically determining the query popularity of each high-frequency query pattern based on the number of matching queries and the latest hit time of each high-frequency query pattern in the high-frequency pattern storage component, and deleting high-frequency query patterns whose query popularity is lower than a preset popularity threshold. This allows for lifecycle management of each high-frequency query pattern in the high-frequency pattern storage component, promptly deleting high-frequency query patterns that no longer frequently appear in log statistics query requests obtained from the data storage device.

[0051] Optionally, for each high-frequency query pattern, the number of matching queries for that pattern can be the total number of times within a unit of time that the high-frequency query pattern is identified as a target high-frequency query pattern matching the query pattern of the log statistics query request. The unit of time can be 1 hour. The latest hit time for the high-frequency query pattern can be the time difference between the current moment and the time when the high-frequency query pattern was last identified as a target high-frequency query pattern matching the query pattern of the log statistics query request. The number of matching queries and the latest hit time for the high-frequency query pattern can be statistically analyzed. The query popularity of a query pattern can be a value determined based on the number of matching queries and the latest hit time, used to measure the frequency of the query pattern appearing in the log statistics query requests obtained by the data storage device. The higher the query popularity, the more frequently the query pattern appears in the log statistics query requests obtained by the data storage device. A query popularity below a preset popularity threshold means that the query popularity is less than the preset popularity threshold. The preset popularity threshold can be a pre-set value. Generally, when the query popularity of a certain high-frequency query pattern is lower than the preset popularity threshold, it indicates that the high-frequency query pattern no longer appears frequently in the log statistics query requests obtained by the data storage device.

[0052] Optionally, at preset time intervals, an operation can be performed to determine the query popularity of each high-frequency query pattern based on the number of matching queries and the latest hit time of each high-frequency query pattern in the high-frequency pattern storage component, and to delete high-frequency query patterns whose query popularity is lower than a preset popularity threshold. The preset time interval can be a pre-set time interval.

[0053] Optionally, the query popularity of each high-frequency query pattern is determined based on the number of matching queries and the latest hit time of each high-frequency query pattern in the high-frequency pattern storage component. This includes performing the following operations for each high-frequency query pattern in the high-frequency pattern storage component: counting the number of matching queries and the latest hit time of the high-frequency query pattern; and using the following query popularity calculation formula, determining the query popularity of the high-frequency query pattern based on the number of matching queries and the latest hit time of the high-frequency query pattern: Where R represents the query popularity of the high-frequency query pattern, P represents the number of matching queries for the high-frequency query pattern, Z represents the latest hit time for the high-frequency query pattern, and a and b are two preset coefficients. The query popularity calculation formula can be a pre-set formula used to calculate the query popularity of a query pattern.

[0054] Optionally, deleting high-frequency query patterns with query popularity lower than a preset popularity threshold includes: determining whether the query popularity of each high-frequency query pattern in the high-frequency pattern storage component is lower than the preset popularity threshold; and deleting high-frequency query patterns in the high-frequency pattern storage component with query popularity lower than the preset popularity threshold.

[0055] Optionally, output the query results of the log statistics query request, including: sending the query results of the log statistics query request to the terminal device of the user who made the query request.

[0056] The technical solution of this invention involves processing each log statistics query request after obtaining query instruction information, determining the query mode of each log statistics query request, continuously counting the number of times the determined query mode is triggered within a preset time window, marking query modes that meet high-frequency conditions as high-frequency query modes, and periodically retrieving business logs that match the high-frequency query modes from the business log library to obtain accelerated aggregation data for the high-frequency query modes. When at least one query mode is marked as a high-frequency query mode, after obtaining a log statistics query request, the query mode of the log statistics query request is determined, and it is detected whether there is a target high-frequency query mode that matches the query mode of the log statistics query request. If a target high-frequency query mode exists, the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query mode, and the business logs in the business log library. This solution addresses the problem in related log statistics query schemes where, after each log statistics query request, the relevant business logs need to be retrieved from a business log database containing a large number of business logs in real time, resulting in long query times and high resource consumption. It automatically identifies and statistically analyzes the query patterns of log statistics query requests. For frequently occurring query patterns, it pre-retrieves business logs matching the query pattern from the business log database. Then, when subsequent log statistics query requests with frequently occurring patterns are received, the query results are determined based on the pre-retrieved business logs and the original business logs in the business log database. This significantly reduces the number of business logs that need to be retrieved from the business log database in real time, reduces query time, lowers resource consumption during the query process, and improves query response speed.

[0057] Example 2 Figure 2 This is a flowchart illustrating a log statistics query method provided in Embodiment 2 of the present invention. This embodiment of the present invention can be combined with various optional solutions from one or more of the above embodiments. For example... Figure 2 As shown, the method includes: Step 201: After obtaining the query instruction information, start processing each log statistics query request, determine the query mode of each log statistics query request, continuously count the number of times the determined query mode is triggered within the preset time window, mark the query mode that meets the high-frequency condition as the high-frequency query mode, and periodically obtain the business logs that meet the high-frequency query mode from the business log library to obtain the accelerated aggregation data of the high-frequency query mode.

[0058] Step 202: When at least one query pattern is marked as a high-frequency query pattern, after obtaining the log statistics query request, determine the query pattern of the log statistics query request and detect whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request.

[0059] Step 203: If a target high-frequency query pattern exists and the target time interval of the log statistics query request includes both aggregated and unaggregated intervals, then determine the query result of the log statistics query request in the aggregated interval based on the accelerated aggregation data of the target high-frequency query pattern.

[0060] Step 204: Based on the business logs in the business log library, determine the query results of the log statistics query request in the non-aggregated range.

[0061] Step 205: Merge the query results of the log statistics query request in the aggregated and unaggregated intervals to obtain the query results of the log statistics query request, and output the query results.

[0062] The technical solution of this invention can, after obtaining a log statistics query request with a frequently occurring query pattern, determine the query result of the log statistics query request based on the pre-acquired business logs and the original business logs in the business log library. This can significantly reduce the number of business logs that need to be obtained from the business log library in real time, reduce query time, reduce resource consumption during the query process, improve query response speed, and enhance the intelligence of the log statistics query process.

[0063] Example 3 Figure 3 This is a schematic diagram of a log statistics query device provided in Embodiment 3 of the present invention. The device can be configured in an electronic device. Figure 3 As shown, the device includes: a pattern processing module 301, a pattern matching module 302, and a pattern query module 303.

[0064] The module 301 is used to process each log statistics query request after obtaining the query instruction information, determine the query mode of each log statistics query request, continuously count the number of times the determined query mode is triggered within a preset time window, mark the query mode that meets the high-frequency condition as a high-frequency query mode, and periodically retrieve business logs that meet the high-frequency query mode from the business log library to obtain accelerated aggregation data of the high-frequency query mode. The module 302 is used to determine the query mode of the log statistics query request after obtaining the log statistics query request when at least one query mode is marked as a high-frequency query mode, and detect whether there is a target high-frequency query mode that matches the query mode of the log statistics query request. The module 303 is used to determine the query result of the log statistics query request based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query mode, and the business logs in the business log library if a target high-frequency query mode exists, and output the query result.

[0065] The technical solution of this invention involves processing each log statistics query request after obtaining query instruction information, determining the query mode of each log statistics query request, continuously counting the number of times the determined query mode is triggered within a preset time window, marking query modes that meet high-frequency conditions as high-frequency query modes, and periodically retrieving business logs that match the high-frequency query modes from the business log library to obtain accelerated aggregation data for the high-frequency query modes. When at least one query mode is marked as a high-frequency query mode, after obtaining a log statistics query request, the query mode of the log statistics query request is determined, and it is detected whether there is a target high-frequency query mode that matches the query mode of the log statistics query request. If a target high-frequency query mode exists, the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query mode, and the business logs in the business log library. This solution addresses the problem in related log statistics query schemes where, after each log statistics query request, the relevant business logs need to be retrieved from a business log database containing a large number of business logs in real time, resulting in long query times and high resource consumption. It automatically identifies and statistically analyzes the query patterns of log statistics query requests. For frequently occurring query patterns, it pre-retrieves business logs matching the query pattern from the business log database. Then, when subsequent log statistics query requests with frequently occurring patterns are received, the query results are determined based on the pre-retrieved business logs and the original business logs in the business log database. This significantly reduces the number of business logs that need to be retrieved from the business log database in real time, reduces query time, lowers resource consumption during the query process, and improves query response speed.

[0066] In an optional embodiment of the present invention, the mode processing module 301, when performing the operation of determining the query mode of each log statistics query request, is specifically used to: for each obtained log statistics query request, determine the query mode of the log statistics query request based on the key query parameters in the log statistics query request.

[0067] In an optional embodiment of the present invention, when the pattern processing module 301 performs the operation of marking the query pattern that meets the high-frequency condition as a high-frequency query pattern, it is specifically used to: determine the query pattern that is triggered more than a preset number of times within a preset time window as a high-frequency query pattern, and store the high-frequency query pattern in the high-frequency pattern storage component.

[0068] In an optional embodiment of the present invention, the pattern processing module 301, when performing the operation of periodically retrieving business logs conforming to high-frequency query patterns from the business log library to obtain accelerated aggregation data of high-frequency query patterns, is specifically configured to: perform the following operations for each high-frequency query pattern: determine the data storage component corresponding to the high-frequency query pattern; periodically retrieve business logs conforming to the high-frequency query pattern from the business log library according to a preset time granularity, and store the retrieved business logs in the data storage component corresponding to the high-frequency query pattern; wherein, the business logs conforming to the high-frequency query pattern stored in the data storage component corresponding to the high-frequency query pattern constitute the accelerated aggregation data of the high-frequency query pattern.

[0069] In an optional embodiment of the present invention, the pattern query module 303, when performing the operation of determining the query result of the log statistics query request based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query pattern, and the business logs in the business log library, specifically performs the following: if the target time interval of the log statistics query request includes an aggregated interval and an unaggregated interval, then, based on the accelerated aggregation data of the target high-frequency query pattern, determine the query result of the log statistics query request in the aggregated interval; based on the business logs in the business log library, determine the query result of the log statistics query request in the unaggregated interval; and merge the query results of the log statistics query request in the aggregated interval and the unaggregated interval to obtain the query result of the log statistics query request.

[0070] In an optional embodiment of the present invention, the pattern query module 303, when performing the operation of determining the query result of the log statistics query request based on the target time interval of the log statistics query request, the accelerated aggregation data of the target high-frequency query pattern, and the business logs in the business log library, is further configured to: if the target time interval of the log statistics query request only includes the aggregated interval, then determine the query result of the log statistics query request based on the accelerated aggregation data of the target high-frequency query pattern.

[0071] In an optional embodiment of the present invention, the log statistics query device may further include: a pattern management module, used to periodically determine the query popularity of each high-frequency query pattern based on the number of matching queries and the latest hit time of each high-frequency query pattern in the high-frequency pattern storage component, and delete high-frequency query patterns whose query popularity is lower than a preset popularity threshold.

[0072] The log statistics query device provided in this embodiment of the invention can execute the log statistics query method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0073] Example 4 Figure 4 A schematic diagram of an electronic device 10, which can be used to implement the log statistics query method of embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0074] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory 12 or a random access memory 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the read-only memory 12 or loaded from storage unit 18 into the random access memory 13. The random access memory 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, read-only memory 12, and random access memory 13 are interconnected via a bus 14. An input / output interface 15 is also connected to the bus 14.

[0075] Multiple components in electronic device 10 are connected to input / output interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of monitors, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as log statistical query methods.

[0077] In some embodiments, the log statistics query method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on a heterogeneous hardware accelerator via read-only memory and / or a communication unit. When the computer program is loaded into random access memory and executed by a processor, one or more steps of the log statistics query method described above may be performed. Alternatively, in other embodiments, the processor may be configured to execute the log statistics query method by any other suitable means (e.g., by means of firmware).

[0078] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices (PLCs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.

[0080] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0081] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).

[0082] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0083] A computing system can include clients and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and establishing a client-electronic device relationship between them. Electronic devices can be cloud electronic devices, also known as cloud computing electronic devices or cloud servers, which are host products within the cloud computing service system. These address the shortcomings of traditional physical hosts and virtual private server services, such as high management difficulty and weak business scalability.

[0084] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log statistics query method, characterized in that, include: After obtaining the query instruction information, the system begins to process each log statistics query request, determines the query mode of each log statistics query request, continuously counts the number of times the determined query mode is triggered within a preset time window, marks the query modes that meet the high-frequency conditions as high-frequency query modes, and periodically retrieves business logs that meet the high-frequency query modes from the business log library to obtain accelerated aggregation data of high-frequency query modes. When at least one query pattern is marked as a high-frequency query pattern, after obtaining the log statistics query request, determine the query pattern of the log statistics query request and check whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request. If a target high-frequency query pattern exists, the query result of the log statistics query request is determined based on the target time interval of the log statistics query request, the accelerated aggregated data of the target high-frequency query pattern, and the business logs in the business log library, and the query result is output.

2. The log statistics query method according to claim 1, characterized in that, Determine the query pattern for each log statistics query request, including: For each log statistics query request obtained, the query mode of the log statistics query request is determined based on the key query parameters in the log statistics query request.

3. The log statistics query method according to claim 1, characterized in that, Query patterns that meet high-frequency conditions are marked as high-frequency query patterns, including: Query patterns that are triggered more than a preset threshold number of times within a preset time window are identified as high-frequency query patterns and stored in the high-frequency pattern storage component.

4. The log statistics query method according to claim 1, characterized in that, Periodically retrieve business logs that match high-frequency query patterns from the business log database to obtain accelerated aggregated data based on these patterns, including: For each high-frequency query pattern, perform the following operations: Identify the data storage components corresponding to the high-frequency query patterns; According to the preset time granularity, business logs that conform to the high-frequency query pattern are periodically retrieved from the business log library, and the retrieved business logs are stored in the data storage component corresponding to the high-frequency query pattern; among them, the business logs that conform to the high-frequency query pattern stored in the data storage component corresponding to the high-frequency query pattern constitute the accelerated aggregation data of the high-frequency query pattern.

5. The log statistics query method according to claim 1, characterized in that, Based on the target time range of the log statistics query request, the accelerated aggregation data of the target high-frequency query pattern, and the business logs in the business log library, determine the query results of the log statistics query request, including: If the target time interval of the log statistics query request includes both aggregated and unaggregated intervals, then the query result of the log statistics query request in the aggregated interval is determined based on the accelerated aggregation data of the target high-frequency query pattern. Based on the business logs in the business log library, determine the query results of the log statistics query request in the non-aggregated range; The query results of the log statistics query request in the aggregated and unaggregated intervals are merged to obtain the query result of the log statistics query request.

6. The log statistics query method according to claim 5, characterized in that, Based on the target time range of the log statistics query request, the accelerated aggregation data of the target high-frequency query pattern, and the business logs in the business log library, the query results of the log statistics query request are determined, including: If the target time interval of the log statistics query request only includes the aggregated interval, the query result of the log statistics query request will be determined based on the accelerated aggregated data of the target high-frequency query pattern.

7. The log statistics query method according to claim 3, characterized in that, Also includes: Based on the number of matching queries and the latest hit time of each high-frequency query mode in the high-frequency mode storage component, the query popularity of each high-frequency query mode is determined periodically, and high-frequency query modes with query popularity lower than a preset popularity threshold are deleted.

8. A log statistics query device, characterized in that, include: The pattern processing module is used to process each log statistics query request after obtaining the query instruction information, determine the query pattern of each log statistics query request, continuously count the number of times the determined query pattern is triggered within a preset time window, mark the query patterns that meet the high-frequency conditions as high-frequency query patterns, and periodically obtain business logs that meet the high-frequency query patterns from the business log library to obtain accelerated aggregation data of high-frequency query patterns. The pattern matching module is used to determine the query pattern of the log statistics query request after obtaining the log statistics query request when at least one query pattern is marked as a high-frequency query pattern, and to detect whether there is a target high-frequency query pattern that matches the query pattern of the log statistics query request. The pattern query module is used to determine the query result of the log statistics query request based on the target time interval of the log statistics query request, the accelerated aggregated data of the target high-frequency query pattern, and the business logs in the business log library, and output the query result if a target high-frequency query pattern exists.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the log statistics query method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log statistics query method according to any one of claims 1-7.

Images