Log analysis method and device based on sampling and readable storage medium

Abstract

The embodiment of the invention provides a sampling-based log analysis method and device and a readable storage medium, after an electronic device obtains an analysis request, a target request number is determined according to a target time period and a target object, and the target request number is used for indicating the number of access requests for accessing the target object in the target time period. And then, the electronic equipment determines a first sampling table from the plurality of sampling tables according to the target request number, and analyzes the first sampling table according to the target dimension to obtain an analysis result. By adopting the scheme, the electronic equipment samples the structured logs according to different sampling rates in a storage stage, and generates and stores the corresponding sampling table; in the analysis stage, the adaptive first sampling table is selected from the multiple sampling tables for analysis according to the query condition, the analysis result is obtained by adaptively selecting the corresponding sampling table based on dynamic sampling storage and log analysis, and the purpose of improving log analysis efficiency and accuracy is achieved.

Description

Sampling-based log analysis methods, devices, and readable storage media Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a sampling-based log analysis method, device, and readable storage medium. Background Technology

[0002] With the rapid development of information technology, enterprises and organizations generate massive amounts of logs in their daily operations. These logs contain a wealth of information that can be used for security analysis, such as identifying attack characteristics and analyzing user behavior.

[0003] Traditional log analysis methods perform full analysis of logs, meaning that no log entries are discarded. The collected logs are fully stored, parsed, and analyzed to output analysis results, such as security vulnerability location and abnormal attack tracing.

[0004] While the log analysis methods described above can provide comprehensive analysis results, the large amount of data leads to long analysis times and heavy resource consumption, making it impossible to meet real-time requirements and greatly limiting the efficiency of security analysis. Summary of the Invention

[0005] This application provides a sampling-based log analysis method, device, and readable storage medium. It selects a suitable sampling table from multiple sampling tables for analysis based on query conditions, which greatly reduces the amount of computation and improves the real-time performance of log analysis.

[0006] In a first aspect, embodiments of this application provide a sampling-based log analysis method, comprising: obtaining an analysis request carrying query conditions, wherein the query conditions at least indicate a target time period, a target object, and a target dimension to be analyzed; determining a target number of requests based on the target time period and the target object, wherein the target number of requests indicates the number of access requests to the target object within the target time period; determining a first sampling table from multiple sampling tables based on the target number of requests, wherein different sampling tables in the multiple sampling tables correspond to different sampling rates, the sampling tables are used to store structured logs requesting the target object under the corresponding sampling rate, and the sampling rate of the first sampling table is negatively correlated with the target number of requests; and analyzing the first sampling table based on the target dimension to obtain analysis results.

[0007] Secondly, embodiments of this application provide a sampling-based log analysis device, comprising: an acquisition module, configured to acquire an analysis request carrying query conditions, wherein the query conditions at least indicate a target time period, a target object, and a target dimension to be analyzed; a first determination module, configured to determine a target request number based on the target time period and the target object, wherein the target request number indicates the number of access requests to the target object within the target time period; a second determination module, configured to determine a first sampling table from multiple sampling tables based on the target request number, wherein different sampling tables in the multiple sampling tables correspond to different sampling rates, the sampling table is used to store structured logs requesting the target object under the corresponding sampling rate, and the sampling rate of the first sampling table is negatively correlated with the target request number; and a processing module, configured to analyze the first sampling table based on the target dimension to obtain analysis results.

[0008] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it causes the electronic device to implement the method described in the first aspect or various possible implementations of the first aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the method described in the first aspect or various possible implementations of the first aspect.

[0010] Fifthly, embodiments of this application provide a computer program product comprising a computing program, wherein when the computer program is executed by a processor, it implements the method described in the first aspect or various possible implementations of the first aspect.

[0011] The sampling-based log analysis method, device, and readable storage medium provided in this application, after obtaining an analysis request that indicates at least the target time period, target object, and target dimension, determines the target request count based on the target time period and target object. This target request count indicates the number of access requests to the target object within the target time period. Then, a first sampling table is determined from multiple sampling tables based on the target request count, and the first sampling table is analyzed according to the target dimension to obtain analysis results. Different sampling tables correspond to different sampling rates, and the sampling tables are used to store structured logs requesting the target object under the corresponding sampling rate. The sampling rate of the first sampling table is negatively correlated with the target request count. This approach samples structured logs at different sampling rates during the database entry stage, generating and storing corresponding sampling tables. During the analysis stage, a suitable first sampling table is selected from multiple sampling tables based on the query conditions for analysis. By dynamically sampling the database and adaptively selecting the corresponding sampling table during log analysis to obtain analysis results, the efficiency and accuracy of log analysis are improved. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 is a schematic diagram of the network structure of the sampling-based log analysis method provided in an embodiment of this application; Figure 2 is a flowchart of the sampling-based log analysis method provided in an embodiment of this application; Figure 3 is a schematic diagram of the log sampling process in the sampling-based log analysis method provided in an embodiment of this application; Figure 4 is another flowchart of the sampling-based log analysis method provided in an embodiment of this application; Figure 5 is a schematic diagram of the sampling-based log analysis device provided in an embodiment of this application; Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] Every time a user visits a website, access logs are generated. With technological advancements, the speed and volume of access log generation are constantly increasing. These logs contain a wealth of information that can be used to identify attack signatures, analyze user behavior, and perform other security analyses. In security analysis scenarios, efficiency and user experience are paramount; efficient analysis tools can significantly shorten attack response times and reduce potential losses.

[0015] Access logs, especially security logs, are typically analyzed using full-volume data analysis. Security logs are a type of access log that specifically records access behaviors related to website security, such as attack attempts and abnormal access. While full-volume data analysis can provide comprehensive results, the sheer volume of data leads to long analysis times, heavy resource consumption, and an inability to meet real-time requirements, significantly limiting the efficiency of security analysis.

[0016] Based on this, embodiments of this application provide a sampling-based log analysis method, device, and readable storage medium. By dynamically sampling and storing logs, and adaptively selecting the corresponding sampling table during log analysis to obtain analysis results, the efficiency and accuracy of log analysis are improved. Specifically, during the storage phase, structured logs are sampled at different sampling rates to generate and store corresponding sampling tables. During the analysis phase, a suitable first sampling table is selected from multiple sampling tables based on query conditions for analysis, significantly reducing computational load while improving the real-time performance of log analysis.

[0017] Figure 1 is a schematic diagram of the network structure of the sampling-based log analysis method provided in an embodiment of this application. Referring to Figure 1, the network architecture includes at least: multiple log sources 11 and a data platform 12, with each log source 11 establishing a network connection with the data platform 12.

[0018] Log source 11 is used to generate raw logs, such as access logs and system logs. Log source 11 is a device or system capable of handling client access requests and forwarding access traffic, including but not limited to application servers, reverse proxy servers, load balancers, content delivery network (CDN) nodes, application programming interface (API) gateways, security gateways, etc.

[0019] The data platform 12 is a platform that integrates hardware resources, software systems, etc. It is often a cluster of multiple servers. As a unified processing hub for log analysis, it is responsible for standardizing and cleaning the raw logs from various log sources 11, converting unstructured raw logs into structured logs, and supporting subsequent business needs such as sampling and analysis.

[0020] Data Platform 12 includes core modules such as a Cloud-Oriented Data Storage (CODS) cluster, a Big Data Query Engine (BQuery), a Message Exchange Service (MES), and a file queue. It also utilizes task components like Flink to implement the storage of sample tables at different sampling rates. The CODS cluster is responsible for parsing and cleaning raw logs. Parsing converts raw logs into structured logs, while cleaning filters out invalid data, such as empty logs and logs with incorrect formats.

[0021] BQuery receives structured logs processed by the CODS cluster and provides high-capacity, high-reliability storage. Furthermore, BQuery splits the structured logs into multiple MES files and maintains a file list, with these multiple MES files forming an MES file queue. Based on BQuery, storage and consumption are decoupled, preventing downstream consumption pressure from directly impacting the storage layer.

[0022] Data Platform 12 uses task components to handle the ingestion of sampled tables with different sampling rates. Taking a sampled table comprising a full table, a table with a 1% sampling rate, and a table with a 10% sampling rate as an example, Data Platform 12 deploys three independent task components, each responsible for ingesting one of the three tables respectively. For each sampled table, different MES files in the MES file queue are assigned to multiple task units within the task components. These task units extract structured logs from the MES files according to the sampling rate, thus achieving efficient ingestion of the sampled table. The task components support stream processing, batch processing, and include, but are not limited to, Flink jobs, Spark jobs, and Storm jobs.

[0023] Figure 2 is a flowchart of a sampling-based log analysis method provided in an embodiment of this application. The execution subject of this embodiment is an electronic device, such as the data platform mentioned above. This embodiment includes: 201, obtaining an analysis request carrying query conditions, wherein the query conditions at least indicate the target time period, target object, and target dimension to be analyzed.

[0024] When log analysis is required, the electronic device receives an analysis request. For example, a staff member enters query conditions such as the target time period, target object, and target dimension on the terminal device. The terminal device generates an analysis request carrying the query conditions based on the staff member's input and sends it to the electronic device.

[0025] In this embodiment of the application, the target time period can be flexibly set. For example, if the current time is 12:00 on December 10th, the target time period is from 11:00 on December 10th to 12:00 on December 10th, which is 1 hour; or if the current time is 12:00 on December 10th, the target time period is from 9:00 on December 10th to 12:00 on December 10th, which is 3 hours; or if the current time is 12:00 on December 10th, the target time period is from 12:00 on December 8th to 12:00 on December 10th, which is 2 days.

[0026] The target can be a domain name, subdomain, etc. For example, a main domain name could be a.com, and subdomains could be www.a.com, tieba.a.com, etc.

[0027] The target dimension can be any one of multiple query dimensions, including but not limited to total number of requests, sample logs, city, Uniform Resource Locator (URL), IP, user agent, Hypertext Transfer Protocol (HTTP) status code, and HTTP referrer.

[0028] 202. Determine the target number of requests based on the target time period and the target object, wherein the target number of requests is used to indicate the number of access requests to the target object within the target time period.

[0029] For example, if the current time is 12:00 PM on December 10th, and the target time period is from 11:00 AM to 12:00 PM on December 10th, which is one hour, the electronic device determines the number of access requests to the target object within this one-hour period. A higher target request count means more access requests to the target object within the target time period. The target request count is an estimated number.

[0030] 203. Determine a first sampling table from multiple sampling tables based on the target number of requests. Different sampling tables in the multiple sampling tables correspond to different sampling rates. The sampling table is used to store structured logs of requests for the target object under the corresponding sampling rate. The sampling rate of the first sampling table is negatively correlated with the target number of requests.

[0031] In this embodiment of the application, different sampling tables in the multiple sampling tables correspond to different sampling rates. For example, the multiple sampling tables include a full table, a sampling table with a sampling rate of 10%, a sampling table with a sampling rate of 1%, a sampling table with a sampling rate of 2%, etc.

[0032] The electronic device determines the first sampling table from multiple sampling tables based on the target number of requests. The larger the target number of requests, the lower the sampling rate of the first sampling table. For example, if the target number of requests is ≥1,500,000, the first sampling table is a sampling table with a sampling rate of 1%; if 1,000,000 ≤ target number of requests ≤1,500,000, the first sampling table is a sampling table with a sampling rate of 10%; and if the target number of requests is <1,000,000, the first sampling table is a sampling table with a sampling rate of 100%, i.e., a full table.

[0033] 204. Analyze the first sampling table according to the target dimension to obtain the analysis results.

[0034] After selecting the first sampling table from multiple sampling tables based on the target request count, the electronic device analyzes the first sampling table in conjunction with the target dimension to obtain the analysis results. For example, the target dimension is the total number of requests. The target request count is an estimated figure used only to determine the first sampling table from multiple sampling tables. The target request count may not be the same as the total number of access requests to the target object within the target time period, and the error may even be significant. Therefore, after determining the first sampling table, the electronic device queries the first sampling table to obtain the number of access requests that match "target object + specified target time period," and returns this number as the analysis result. Assume the target number of requests is 2 million, the target time period is from 10:00 AM on December 3rd to 10:00 AM on December 5th, a time span of 2 days. The sampling rate of the first sampling table is 10%, so the table lookup yields 1 million. Since the sampling rate is 10%, 1 million × 10 = 10 million, meaning the actual total number of requests is 10 million.

[0035] For example, the target dimension is sample logs. Assuming the target time period is from 10:00 AM on December 3rd to 10:00 AM on December 5th, and the sampling rate of the first sampling table is 10%, a lookup of the table yields a total of 20,000 structured log entries. If it is pre-set that only 10 of these logs will be selected as sample logs, then the electronic device will randomly select 10 logs from the 20,000 structured logs or select 10 structured logs according to certain rules as sample logs.

[0036] For example, the target dimension is URLs. Assume the target time period is from 10:00 AM on December 3rd to 10:00 AM on December 5th. The sampling rate of the first sampling table is 10%, and the table shows a total of 100 unique URLs. The electronic device counts the number of access requests for each URL, sorts them in descending order of quantity, and returns the top 10 URLs. If the table shows that the number of URLs is less than or equal to 10, all URLs are returned.

[0037] The sampling-based log analysis method provided in this application involves an electronic device receiving an analysis request that specifies at least the target time period, target object, and target dimension. The electronic device then determines the target request count based on the target time period and target object, indicating the number of access requests to the target object within the target time period. Next, the electronic device determines a first sampling table from multiple sampling tables based on the target request count and analyzes the first sampling table according to the target dimension to obtain analysis results. Different sampling tables correspond to different sampling rates, and the sampling tables store structured logs requesting the target object at the corresponding sampling rate. The sampling rate of the first sampling table is negatively correlated with the target request count. Using this approach, the electronic device samples structured logs at different sampling rates during the data entry stage, generating and storing corresponding sampling tables. During the analysis stage, a suitable first sampling table is selected from multiple sampling tables based on the query conditions for analysis. By dynamically sampling the data entry and adaptively selecting the corresponding sampling table during log analysis to obtain analysis results, the efficiency and accuracy of log analysis are improved.

[0038] In step 204 of Figure 2 above, the electronic device analyzes the first sampling table in different ways depending on the target dimension. For example, for the total number of requests, sample logs, cities, etc., only the first sampling table needs to be queried to obtain the analysis results. In these cases, the first sampling table is also called the query table; or, the method of using the first sampling table is called the query table.

[0039] For example, to obtain the analysis results for the top URLs, top IPs, etc., the first sampling table needs to be aggregated and calculated, and cannot be obtained by directly looking up the table. In this case, the first sampling table is also called a ranking table; or, the method of using the first sampling table is called a ranking table.

[0040] Optionally, when the analysis results for the target dimension require aggregation calculations on the first sampling table, the electronic device further determines the deduplicated independent number of the target object in the target dimension from the independent number cache, and determines whether the first sampling table needs to be replaced with the second sampling table based on the deduplicated independent number of the target dimension. When it is necessary to replace the first sampling table with the second sampling table, the electronic device analyzes the second sampling table according to the target dimension to obtain the analysis results; when it is not necessary to replace the first sampling table with the second sampling table, the electronic device analyzes the first sampling table according to the target dimension to obtain the analysis results. The independent number cache stores the independent number corresponding to each query dimension. The independent number under the target dimension refers to the total number of unique target dimensions after deduplication among all access requests that access the target object within a first time period. The sampling rate of the second sampling table is lower than the sampling rate of the first sampling table.

[0041] For example, in this embodiment of the application, the electronic device maintains two caches: an independent number cache and a total request count cache. The independent number cache stores the independent numbers corresponding to different query dimensions for each analysis object. For example, if the analysis object is www.a.com, the independent number cache stores the independent number of URLs, the independent number of IPs, etc. The independent number of URLs is the number of unique URLs after deduplication among the access requests to www.a.com within the first time period; the independent number of IPs is the number of unique IPs after deduplication among the access requests to www.a.com within the first time period; and the independent number of cities is the number of unique cities after deduplication of the cities where the IP addresses are located among the access requests to www.a.com within the first time period.

[0042] When the analysis results for the target dimension are aggregated, the electronic device determines the number of unique URLs for the target object after deduplication within the target dimension from the uniqueness cache. For example, if the target dimension is a URL, the electronic device retrieves the number of unique URLs after deduplication from the access requests that accessed the target object within a first time period from the uniqueness cache. Then, the electronic device compares the uniqueness count with a preset parameter. If the uniqueness count is greater than the preset parameter, the electronic device switches from the first sampling table to the second sampling table, i.e., upgrades the sampling level. After upgrading, the electronic device analyzes the second sampling table according to the target dimension to obtain the analysis results.

[0043] Taking an example with 12,000 unique URLs, a preset parameter of 10,000, and a sampling rate of 10% for the first sampling table, since 12,000 > 10,000, the electronic device switches from the first sampling table to the second sampling table, with a sampling rate of 1%. The electronic device then analyzes the second sampling table based on the URLs to obtain the analysis results. For example, if the query condition specifies retrieving the top 10 URLs, the electronic device queries the second sampling table and obtains 20 unique URLs. The electronic device counts the number of access requests for each URL, sorts them in descending order of quantity, and returns the top 10 URLs.

[0044] It should be noted that if the first sampling table has the lowest sampling rate among the multiple sampling tables, even if the electronic device determines based on independent data that it needs to switch the first sampling table to the second sampling table, it cannot continue to switch; instead, the first sampling table remains unchanged. For example, if the sampling tables include a full table, a sampling table with a sampling rate of 10%, and a sampling table with a sampling rate of 1%, and the first sampling table has a sampling rate of 1%, even if an upgrade is needed, the first sampling table will remain unchanged because its sampling rate is already the lowest.

[0045] If the number of unique URLs is 980, since 980 < 10000, the electronic device does not need to switch the first sampling table to the second sampling table. Instead, it analyzes the first sampling table according to the target dimension to obtain the analysis results.

[0046] In the above embodiments, the first duration is, for example, 24 hours, 12 hours, 20 hours, etc., and this application embodiment is not limited.

[0047] By adopting this approach, when the number of independent objects in the target dimension is relatively high, the electronic device adaptively uses a sampling table with a lower sampling rate. This not only improves the log analysis effect by reducing the number of log analyses, but also ensures that the obtained analysis results are representative, while ensuring the accuracy and reliability of the analysis results.

[0048] Optionally, in the process of determining the target request count based on the target time period and the target object, the electronic device determines the daily request count of the target object from the total request count cache, and determines the target request count based on the daily request count and the target time period. The daily request count of the target object indicates the number of access requests to the target object within the most recent first time period.

[0049] For example, the electronic device maintains two caches: an independent number cache and a total request count cache. The independent number cache stores the independent numbers corresponding to different query dimensions for each analysis object. The total request count cache stores the total number of access requests for each analysis object within the most recent first time period. For example, if the first time period is 24 hours, and the current time is 8:00 AM on December 2nd, and the analysis objects include www.a.com and www.b.com, the total request count cache stores the daily request count for www.a.com and the total request count for www.b.com within the 24 hours from 7:00 AM on December 1st to 7:00 AM on December 2nd. After the electronic device receives an analysis request with query conditions, it determines the target request count based on the target time period in the query conditions and the daily request count in the total request count cache.

[0050] For example, if the target object is www.a.com, the daily request count for www.a.com in the cache is 600,000, the target time period is from 7:00 AM on November 30th to 7:00 AM on December 2nd, and the time span is 2 days, then the target request count = 600,000 × 2 = 1,200,000.

[0051] For example, if the target object is www.b.com, the daily request count for www.b.com in the cache is 600,000, the target time period is from 5:00 AM to 7:00 AM on December 2nd, and the time span is 2 hours, then the target request count = 600,000 × 2 ÷ 24 = 50,000.

[0052] Using this approach, electronic devices can quickly estimate the target number of requests to access the target object within the target time period based on the daily request count cached in the total request count and the duration of the target time period. The calculation logic is simple, efficient, and low-cost.

[0053] The sampling-based log analysis method provided in this application includes an ingestion stage and an analysis stage. In the ingestion stage, the electronic device samples structured logs at different sampling rates, generating and storing corresponding sampling tables. In the analysis stage, given that multiple sampling tables are ingested, the electronic device selects the appropriate first sampling table for analysis based on query conditions. That is, in the ingestion stage, the electronic device adopts a dynamic sampling allocation strategy, dynamically distributing the original logs to sampling tables with different sampling rates based on the request volume in a single batch, rather than uniformly ingesting them proportionally. For example, with sampling rates of 100%, 10%, and 1%, the probability of the same original log corresponding to a structured log appearing in all three tables simultaneously is extremely low, only 10% × 1% = 0.1%. This achieves both ingestion of the three sampling tables and balances data representativeness and storage efficiency through differentiated allocation. In the analysis stage, a query drift mechanism is used to select a suitable first sampling table for analysis based on the input query conditions, thereby reducing computational load and improving log analysis efficiency. In addition, electronic devices regularly cache and update statistics on the daily number of domain requests and other key fields to further optimize analysis performance.

[0054] The above focuses on the analysis phase. Below, we will explain the warehousing phase in detail.

[0055] During the data entry phase, electronic devices use a cluster to collect and dynamically sample massive amounts of raw logs in real time. The data entry phase consists of two sub-phases: raw log collection and parsing, and task component sampling and data entry operations.

[0056] First, the raw logs are collected and parsed.

[0057] Optionally, the electronic device collects raw logs from multiple log sources and parses them to obtain structured logs. The raw logs at least contain logs corresponding to access requests to the target object. The electronic device then uses these structured logs to generate a MES file queue, where each MES file in the queue contains multiple structured logs. After generating the MES file queue, during the sampling and data entry process, the task component samples the structured logs contained in each MES file in the MES file queue according to different sampling rates, resulting in multiple sampling tables.

[0058] For example, a collection component is set up on each log source. The collection component collects raw logs from the log source and sends the collected raw logs to a CODS cluster on the electronic device. The CODS cluster is responsible for parsing and cleaning the raw logs to generate structured logs. The CODS cluster writes the structured logs to BQuery on the electronic device. After receiving the structured logs from CODS, BQuery generates multiple MES files, which form an MES file queue. This MES file queue is consumed by subsequent task components. An MES file queue contains multiple MES files, and an MES file contains multiple structured logs, with one raw log corresponding to one structured log.

[0059] This application does not limit the method of generating MES files. For example, when the accumulated structured logs reach a preset size, an MES file is automatically generated. The preset size is, for example, 1 gigabyte (GB) or 500 megabytes (MB). Alternatively, at fixed intervals, Bquery aggregates the structured logs within that period into an MES file. The fixed intervals are, for example, 1 minute or 5 minutes. This application does not limit the specific intervals.

[0060] This approach parses raw logs into structured logs for easier and more precise querying and analysis. It also enables efficient management of massive log volumes by generating MES files in batches, ensuring the orderliness and stability of log collection. Furthermore, it decouples storage and consumption, preventing downstream consumption pressure from directly impacting the storage layer.

[0061] Secondly, the task component sampling and database entry operation.

[0062] Optionally, in the above embodiments, during the process of the electronic device sampling the structured logs contained in each MES file in the MES file queue according to different sampling rates to obtain multiple sampling tables, for each sampling rate, the electronic device uses the task component corresponding to the sampling rate to read the MES files in the MES file queue. Then, the electronic device assigns each MES file to different task units of the task component and controls each task unit to consume the MES files in parallel according to the sampling rate to generate the sampling table.

[0063] In this embodiment, after BQuery generates the MES file queue, a task component running on the electronic device consumes the MES files in the MES file queue. This task component can be a Flink job. Taking multiple sampling tables, including a sampling table with a sampling rate of 100%, a sampling table with a sampling rate of 10%, and a sampling table with a sampling rate of 1%, as an example, to avoid interference between jobs with different sampling rates when entering the database, sampling tables with different sampling rates are entered into the database by different task components.

[0064] Figure 3 is a schematic diagram of the log sampling process in the sampling-based log analysis method provided in this application embodiment. Referring to Figure 3, taking a Flink job as an example, for each sampling rate, the corresponding task component has multiple task units, such as task unit 1, task unit 2, task unit 3... task unit N, where N is an integer greater than 3. The Job Manager is the manager of the task components. The entry of a sampling table into the database is completed collaboratively by multiple task units of the task component, scheduled by the Job Manager. Taking the entry of a sampling table with a sampling rate of 10% as an example, the Job Manager reads the MES file queue sequentially, assigns different MES files to different task units, and multiple task units consume MES files in parallel. Each task unit parses the MES file, reads the structured logs in the MES file line by line, and samples the structured logs contained in the MES file according to the 10% sampling rate before writing them into the sampling table with a sampling rate of 10%.

[0065] This approach allows sampling tables with different sampling rates to be stored in the database by different task components, preventing interference between the storage of sampling tables with different sampling rates. Furthermore, for each sampling rate, multiple task units within the task component consume the MES file in parallel, thereby improving the processing efficiency of massive logs.

[0066] Optionally, in the above embodiments, during the process of the electronic device controlling each task unit to consume the MES file in parallel according to the sampling rate to generate the sampling table, for each task unit, the electronic device uses the task unit to read each structured log in the MES file to obtain the business time and analysis object corresponding to the structured log. Each time the business time and analysis object corresponding to the structured log are read, the electronic device maintains a request count counter based on the business time and analysis object of the structured log. Different request count counters are used to indicate the number of times different analysis objects are accessed per minute. Then, the electronic device generates the sampling table based on the request count counter, the sampling rate corresponding to the task component to which the task unit belongs, and the sampling effectiveness count. The sampling effectiveness count is used to indicate the triggering condition for sampling according to the sampling rate.

[0067] In this embodiment, the task unit reads the structured logs in the MES file line by line. For each structured log entry, the task unit deserializes the log to obtain the business time and the object of analysis. The business time refers to the generation time of the original log corresponding to the structured log, and the object of analysis includes domain names, subdomains, etc. The task unit maintains the number of times the object of analysis is accessed at a minute granularity. For example, for www.a.com, the request count counter 'a' at 5:44 is 1000, indicating that there were 1000 access requests to www.a.com within that minute; the request count counter 'b' at 5:45 is 50000, indicating that there were 50000 access requests to www.a.com within that minute.

[0068] For example, for www.b.com, the request count counter c at 5:44 is 2000, indicating that there were 2000 access requests to www.b.com in that minute; the request count counter d at 5:45 is 50000, indicating that there were 50000 access requests to www.b.com in that minute.

[0069] Each time the business time and analysis object corresponding to the structured log are determined, for example, at 5:46, the electronic device maintains a request count counter based on the business time and analysis object. Continuing with the above example, assuming the analysis object is www.a.com and the business time is 5:44, then the request count counter 'a' is incremented by 1, making the count of request count counter 'a' 1001. As another example, at 5:46, the electronic device deserializes the structured log and finds that the analysis object is www.b.com and the business time is 5:45, then the request count counter 'd' is incremented by 1, making the count of request count counter 'd' 50001.

[0070] Using this approach, electronic devices determine whether structured logs should be stored based on the number of requests, sampling rate, and sampling effectiveness count. This enables refined dynamic sampling based on the analysis object and business time, ensuring that the sampling strategies for different analysis objects and different time periods are independent and controllable, while taking into account both the representativeness and efficiency of the stored data.

[0071] Optionally, in the above embodiments, during the process of generating the sampling table based on the request count counter, the sampling rate corresponding to the task component to which the task unit belongs, and the sampling effectiveness count, after each maintenance of the request count counter, when the count of the request count counter is less than or equal to the sampling effectiveness count, the electronic device writes the structured log into the sampling table corresponding to the sampling rate; when the count of the request count counter is greater than the sampling effectiveness count, the electronic device determines whether to write the structured log into the sampling table based on the count of the request count counter and the reciprocal of the sampling rate.

[0072] For example, each task component's sampling and data entry job is configured with two parameters: the reciprocal of the sampling rate, S, and the sampling effectiveness count, A. For each structured log entry in the MES file, the task unit maintains a request count counter for the analyzed object based on the analysis object and business time. The electronic device then determines the relationship between the request count counter, C, and the sampling effectiveness count, A. The relationship is as follows: Case 1: C > A.

[0073] Sampling is triggered when the request count counter exceeds the sampling effective count. The electronic device checks if the result of C mod S is equal to 0. When C mod S = 0, the electronic device writes the structured log entry to the sampling table. When C mod S ≠ 0, the electronic device discards the structured log entry and does not perform subsequent database entry processing.

[0074] Case 2: C≤A.

[0075] When the number of requests counters is less than the number of times sampling is effective, sampling is not triggered, and the electronic device writes the structured log entry into the sampling table.

[0076] In this embodiment, the sampling validity count A is configurable to avoid invalid sampling of domains with low traffic and to ensure that the sampled data is representative.

[0077] This approach uses a sampling effectiveness count to ensure that sampling is only applied to logs of target objects accessed in large batches. Logs of target objects with fewer requests are not sampled but are instead fully stored in the database, thus avoiding distortion caused by sampling.

[0078] As described above, for each analysis object, the task unit of the task component running on the electronic device maintains the number of times the analysis object is accessed on a minute-by-minute basis. For example, for www.a.com, the request count counter 'a' at 5:44 is 1000, indicating that there were 1000 access requests to www.a.com within that minute. Below, using the target object as an example, we will explain in detail the activation and expiration of the request count counter.

[0079] Optionally, the electronic device determines the number of raw logs generated from accessing the target object within different minutes based on raw logs collected from multiple log sources. Then, the electronic device determines a target minute based on the number of raw logs generated from accessing the target object within different minutes, and starts the request count counter for that target minute. The target minute has the highest number of raw logs generated from accessing the target object compared to other minutes within the same timeframe. The request counter is used to maintain the number of raw logs generated from accessing the target object within a valid window, where the valid window is a duration centered on the target minute.

[0080] Taking www.a.com as the target object as an example, the task unit counts the number of access requests for each analyzed object within the same minute. Assume there is no 5:46 time period, and therefore no request counter for www.a.com at 5:44. At 5:46, the electronic device uses the task unit to deserialize a batch of structured logs and finds that: at 5:43, due to a large number of users accessing www.a.com, 1000 raw log entries were generated, corresponding to 1000 structured log entries. Similarly, the raw log entries at 5:44 are 100,000, and the raw log entries at 5:45 are 50,000. Among the three times 5:43, 5:44, and 5:45, the number of raw log entries generated at 5:44 due to accessing www.a.com is the highest. Therefore, the electronic device uses 5:44 as the target minute and starts a request counter with a count of 100,000. The effective window for this request counter is 5:14-6:14. Within this hour, if a structured log corresponding to the original log generated at 5:44 and accessing www.a.com is parsed again, the request count counter is incremented by 1 until 6:14. For example, if a structured log corresponding to the original log generated at 5:44 and accessing www.a.com is parsed at 6:10, the count is incremented by 1. After that, the request count counter is invalidated. Additionally, if an original log is outside the valid window, it is fully entered into the database. For example, if 5 structured logs from 5:10 are received at 5:44, these 5 structured logs are fully entered into the database without triggering sampling.

[0081] After the task unit determines the valid window for the request counter of www.a.com, it clears other request counters outside the valid window for www.a.com. For example, if the request counter for www.a.com is 100,000 at 5:10, this counter is cleared. However, if the request counter for www.a.com is 1,000 at 5:20, it is retained because it is within the valid window. In this way, for each domain, there are a maximum of 60 request counters, meaning one request counter per minute within the valid window.

[0082] This approach, by setting a valid window for the request counter, retains only the request counter within the valid window, thus preventing the memory occupied by the request counter from continuously expanding, avoiding electronic device crashes, and ensuring the smooth progress of log analysis.

[0083] Optionally, in the above embodiments, the electronic device updates the daily request count of each analysis object in the daily request count cache every second time interval according to the request count counters, and updates the independent number corresponding to each aggregation dimension in the independent number cache every second time interval according to the request count counters.

[0084] For example, the second duration is 1 hour, and the daily request count is updated every hour, so that the total number of requests is the number of access requests to the analyzed object in the most recent day. If there are no access requests to a certain analyzed object in the most recent day, the daily request count is not updated, and the last statistical result is retained.

[0085] For example, from 7:00 AM on December 2nd to 7:00 AM on December 3rd, www.a.com had 1 million daily requests. When updating the daily request count at 8:00 AM on December 3rd, 10,000 requests from 7:00 AM to 8:00 AM on December 2nd were removed, and 12,000 requests from 7:00 AM to 8:00 AM on December 3rd were added. The updated daily request count is 1 million - 10,000 + 12,000 = 1,002,000. The corresponding window is from 8:00 AM on December 2nd to 8:00 AM on December 3rd.

[0086] For example, from 7:00 AM on December 2nd to 7:00 AM on December 3rd, www.a.com had 1 million daily requests. During the update of the daily request count at 8:00 AM on December 3rd, if the number of requests from 7:00 AM to 8:00 AM on December 2nd was 10,000, and the number of requests from 7:00 AM to 8:00 AM on December 3rd was 0, then the updated daily request count would be 1 million - 10,000 + 0 = 990,000. The corresponding window is from 8:00 AM on December 2nd to 8:00 AM on December 3rd.

[0087] For example, from 7:00 AM on December 1st to 7:00 AM on December 2nd, the daily request count for www.a.com was 5000. From 7:00 AM on December 2nd to 7:00 AM on December 3rd, the number of access requests for www.a.com was 0. During the update of the daily request count at 8:00 AM on December 3rd, since there were no requests in the past day, the daily request count was not updated, and the daily request count remained at 5000.

[0088] Similarly, the uniqueness count in the uniqueness cache for each aggregation dimension is updated every hour, ensuring that the uniqueness count reflects the number of unique URLs accessed within the analysis object within the target dimension in the most recent day. For example, for the analysis object www.a.com, there were 9500 unique URLs after deduplication from 7:00 AM on December 2nd to 7:00 AM on December 3rd (24 hours). Then, 300 new URLs appeared in the hour from 7:00 AM to 8:00 AM on December 3rd, and another 200 URLs appeared in the hour from 7:00 AM on December 2nd to 8:00 AM on December 3rd. Therefore, the updated uniqueness count is 9500 + 300 - 200 = 9600.

[0089] This approach automatically refreshes the daily request count for analyzed objects in the total request count cache and the independent count for each aggregation dimension in the independent count cache every hour, ensuring data timeliness and accuracy. Furthermore, for analyzed objects with no requests, the daily request count in the total request count cache remains unchanged, preventing data loss.

[0090] As described in the above embodiments, after the electronic device generates a list of MES files using structured logs, it samples the structured logs contained in each MES file in the MES file queue according to different sampling rates, obtaining multiple sampling tables. Simultaneously, the electronic device maintains an independent data cache and a total request cache. During subsequent log analysis, the first sampling table is determined based on the daily request count in the total request cache. When the analysis results for the target dimension require aggregation calculation, the system further determines whether to use a sampling table with a lower sampling rate, i.e., whether to upgrade, based on the independent data count of the target dimension in the independent data cache. For example, please refer to Table 1.

[0091] Table 1

[0092] Table 1 illustrates the total request cache and the unique request cache, using the domain name as the target object as an example.

[0093] In this embodiment, the target dimension is any one of multiple query dimensions. There are two types of query dimensions: one type requires no aggregation calculation to obtain the analysis results, but can be obtained directly by querying the first sampling table; in this scenario, the sampling table is called the query table. The other type requires aggregation calculation to obtain the analysis results, i.e., performing statistics on the first sampling table; in this scenario, the sampling table is called the top table. See Table 2 for examples.

[0094] Table 2

[0095] Please refer to Table 2. Electronic devices should query Table 2 based on the target dimension to determine whether to use a query table or a toptable. When using a query table, the analysis results can be obtained simply by looking up the table; when using a toptable, aggregation calculations need to be performed on the sample table. The city can use either a query table or a toptable, depending on the requirements; this application does not impose any restrictions.

[0096] The following example, using www.a.com as the target website, with 600,000 daily requests from 11:00 AM on December 4th to 12:00 PM on December 5th, 980 unique requests per city, 12,000 unique requests per URL, and 8,000 unique requests per IP, demonstrates the sampling-based log analysis method described above. The method is pre-configured with the total number of requests, using a query table for sample logs, and a top table for city, URL, and IP. For an example, please refer to Figure 4.

[0097] Figure 4 is another flowchart of the sampling-based log analysis method provided in this embodiment. This embodiment includes: 401, receiving an analysis request carrying query conditions.

[0098] This analysis request includes query criteria, specifying the target time period as December 3rd, 10:00 AM to December 5th, 10:00 AM. Target dimensions are total request count, sample logs, top cities, top URLs, or top IPs.

[0099] 402. Parse the query conditions to obtain the target object, target time period, and target dimension.

[0100] 403. Determine the target number of requests based on the target time period and target object.

[0101] Since the total number of requests in the cache is 600,000 per day, and the target time period spans 2 days, the target number of requests is 600,000 × 2 = 1,200,000.

[0102] When the target dimension is Top city, Top URL, or Top IP, the electronic device also queries the unique count cache to determine the unique counts for each of Top city, Top URL, and Top IP. Top city: 980, URL: 12000, IP: 8000.

[0103] 404. Determine the first sampling table based on the target number of requests.

[0104] Since the target number of requests, 1,200,000, satisfies the condition: 1,000,000 ≤ target number of requests ≤ 1,500,000, the first sampling table is a sampling table with a sampling rate of 10%.

[0105] If the target number of requests is greater than or equal to 1,500,000, then the first sampling table is a sampling table with a sampling rate of 1%; if the target number of requests is less than 1,000,000, then the first sampling table is a sampling table with a sampling rate of 100%, i.e., a full table.

[0106] 405. Determine whether aggregation calculation is needed for the first sampling table. If aggregation calculation is needed for the first sampling table, proceed to step 406; if aggregation calculation is not needed for the first sampling table, proceed to step 408.

[0107] When the target dimension is the total number of requests or sample logs, a query table is used, and there is no need to perform aggregation calculations on the first sampling table. Therefore, step 408 is executed to obtain the analysis results. When the target dimensions are Top city, Top URL, and Top IP, a Top table is used, and aggregation calculations on the first sampling table are required. Therefore, step 406 is executed to determine whether to upgrade the first sampling table.

[0108] 406. Determine whether the number of independent numbers is greater than the preset parameter. If the number of independent numbers is greater than the preset parameter, proceed to step 407; if the number of independent numbers is less than or equal to the preset parameter, proceed to step 408.

[0109] Taking a preset parameter of 10000 as an example, when the target dimension is Top city, 980 < 10000, so no upgrade is triggered. When the target dimension is Top URL, 12000 > 10000, so an upgrade is triggered, and the first sampling table is switched to the second sampling table. The sampling rate of the first sampling table is, for example, 10%, and the sampling rate of the second sampling table is, for example, 1%.

[0110] When the target dimension is Top IP, 8000 < 10000, no upgrade is triggered.

[0111] 407. Analyze the second sampling table according to the target dimension to obtain the analysis results.

[0112] 408. Analyze the first sampling table according to the target dimension to obtain the analysis results.

[0113] Based on the above, we can obtain the following Table 3.

[0114] Table 3. Query Dimensions: Table Type, Sampling Rate, Total Requests, Query Table 10%, Sample Logs, Query Table 10%, Citytop Table 10%, URLtop Table 1%, IPtop Table 10%. Assuming the electronic device determines the sampling rate of the first sampling table to be 10% based on the target request count, the analysis results are obtained using the first sampling table when the target dimension is the total number of requests, sample logs, Top City, or Top IP; and when the target dimension is the Top URL, the analysis results are obtained using the second sampling table. The sampling rate of the second sampling table is 1%.

[0115] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0116] Figure 5 is a schematic diagram of a sampling-based log analysis device provided in an embodiment of this application. The sampling-based log analysis device 500 is integrated on an electronic device and includes: an acquisition module 51, a first determination module 52, a second determination module 53, and a processing module 54.

[0117] The acquisition module 51 is used to acquire an analysis request carrying query conditions, wherein the query conditions at least indicate the target time period, target object, and target dimension to be analyzed; the first determination module 52 is used to determine the target request number based on the target time period and the target object, wherein the target request number is used to indicate the number of access requests to the target object within the target time period; the second determination module 53 is used to determine a first sampling table from multiple sampling tables based on the target request number, wherein different sampling tables in the multiple sampling tables correspond to different sampling rates, and the sampling table is used to store structured logs of requests to the target object under the corresponding sampling rate, wherein the sampling rate of the first sampling table is negatively correlated with the target request number; and the processing module 54 is used to analyze the first sampling table based on the target dimension to obtain analysis results.

[0118] In one feasible implementation, the processing module 54 is configured to, when the aggregation calculation is performed to obtain the analysis results of the target dimension, determine the number of unique objects of the target object after deduplication in the target dimension from the unique number cache. The unique number cache stores the unique number corresponding to each query dimension. The unique number under the target dimension refers to the total number of unique target dimensions that are not repeated after deduplication among all access requests that access the target object within a first time period. Based on the number of unique objects after deduplication in the target dimension, determine whether the first sampling table needs to be replaced with a second sampling table, where the sampling rate of the second sampling table is lower than that of the first sampling table. When it is necessary to replace the first sampling table with the second sampling table, analyze the second sampling table according to the target dimension to obtain the analysis results.

[0119] In one feasible implementation, the first determining module 52 is used to determine the daily request count of the target object from the total request count cache based on the target object. The total request count cache stores the daily request count of each analysis object. The daily request count of the target object is used to indicate the number of access requests to the target object within a first time period. The target request count is determined based on the daily request count and the target time period.

[0120] In one feasible implementation, before the second determining module 53 determines the first sampling table from multiple sampling tables based on the target request number, the processing module 54 is further configured to collect raw logs from multiple log sources, the raw logs containing at least logs corresponding to access requests to the target object; parse the raw logs to obtain structured logs; generate an MES file queue using the structured logs, where one MES file in the MES file queue contains multiple structured logs; and sample the structured logs contained in each MES file in the MES file queue according to different sampling rates to obtain multiple sampling tables.

[0121] In one feasible implementation, during the process of the processing module 54 sampling the structured logs contained in each MES file in the MES file queue according to different sampling rates to obtain multiple sampling tables, for each sampling rate, the task component corresponding to the sampling rate reads the MES file in the MES file queue; each MES file is assigned to different task units of the task component; and each task unit is controlled to consume the MES file in parallel according to the sampling rate to generate the sampling table.

[0122] In one feasible implementation, during the process of the processing module 54 controlling each task unit to consume the MES file in parallel according to the sampling rate to generate the sampling table, for each task unit, the task unit reads each structured log in the MES file to obtain the business time and analysis object corresponding to the structured log; after determining the business time and analysis object corresponding to the structured log each time, a request count counter is maintained according to the business time and analysis object, and different request count counters are used to indicate the number of times different analysis objects are accessed per minute; the sampling table is generated according to the request count counter, the sampling rate corresponding to the task component to which the task unit belongs, and the sampling effectiveness count, and the sampling effectiveness count is used to indicate the triggering condition for sampling according to the sampling rate.

[0123] In one feasible implementation, during the process of generating the sampling table based on the request count counter, the sampling rate corresponding to the task component to which the task unit belongs, and the sampling effectiveness count, the processing module 54, after maintaining the request count counter each time, writes the structured log into the sampling table corresponding to the sampling rate when the count of the request count counter is less than or equal to the sampling effectiveness count; and when the count of the request count counter is greater than the sampling effectiveness count, determines whether to write the structured log into the sampling table based on the count of the request count counter and the reciprocal of the sampling rate.

[0124] In one feasible implementation, after the processing module 54 reads the business time and analysis object corresponding to the structured log each time, before maintaining the request count counter based on the business time and analysis object of the structured log, it also determines the number of raw logs generated due to accessing the target object within different minutes based on the raw logs collected from multiple log sources; based on the number of raw logs generated due to accessing the target object within different minutes, it determines a target minute, which has the highest number of raw logs generated due to accessing the target object compared to other minutes within different minutes; and it starts the request count counter for the target minute, which is used to maintain the number of raw logs generated by accessing the target object within a valid window, where the valid window is a period of time centered on the target minute.

[0125] In one feasible implementation, the processing module 54 is further configured to update the daily request count of each analysis object in the total request count cache every second time interval according to the request count counters of each object; and update the independent number corresponding to each aggregation dimension in the independent number cache every second time interval according to the request count counters of each object.

[0126] The sampling-based log analysis device provided in this application embodiment can perform the actions of the electronic devices in the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0127] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 6, the electronic device 600 includes: a processor 61 and a memory 62; the memory 62 stores computer instructions; the processor 61 executes the computer instructions stored in the memory 62, causing the processor 61 to execute the sampling-based log analysis method implemented by the above-mentioned electronic device.

[0128] The specific implementation process of processor 61 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0129] Optionally, the electronic device 600 also includes a communication component 63. The processor 61, memory 62, and communication component 63 can be connected via a bus 64.

[0130] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the sampling-based log analysis method implemented by the above-mentioned electronic device.

[0131] This application also provides a computer program product comprising a computer program that, when executed by a processor, implements the sampling-based log analysis method implemented by the aforementioned electronic device.

[0132] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0133] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A sampling-based log analysis method, characterized in that, include: Obtain an analysis request carrying query conditions, wherein the query conditions at least indicate the target time period, target object, and target dimension to be analyzed; The target number of requests is determined based on the target time period and the target object, wherein the target number of requests is used to indicate the number of access requests to the target object within the target time period; A first sampling table is determined from multiple sampling tables based on the target number of requests. Different sampling tables correspond to different sampling rates. The sampling table is used to store structured logs of requests for the target object under the corresponding sampling rate. The sampling rate of the first sampling table is negatively correlated with the target number of requests. The first sampling table is analyzed according to the target dimension to obtain the analysis results.

2. The method according to claim 1, characterized in that, The step of analyzing the first sampling table according to the target dimension to obtain the analysis result includes: when the demand for obtaining the analysis result of the target dimension is aggregated, determining the number of unique objects of the target object after deduplication in the target dimension from the unique number cache, wherein the unique number cache stores the unique number corresponding to each query dimension, and the unique number under the target dimension refers to the total number of unique target dimensions after deduplication among all access requests that access the target object within a first time period; determining whether the first sampling table needs to be replaced with a second sampling table based on the number of unique objects after deduplication in the target dimension, wherein the sampling rate of the second sampling table is lower than that of the first sampling table; when the first sampling table needs to be replaced with the second sampling table, analyzing the second sampling table according to the target dimension to obtain the analysis result.

3. The method according to claim 1, characterized in that, The step of determining the target request count based on the target time period and the target object includes: determining the daily request count of the target object from the total request count cache based on the target object, wherein the total request count cache stores the daily request count of each analyzed object, and the daily request count of the target object is used to indicate the number of access requests to the target object within a first time period; and determining the target request count based on the daily request count and the target time period.

4. The method according to any one of claims 1 to 3, characterized in that, Before determining the first sampling table from multiple sampling tables based on the target number of requests, the method further includes: collecting raw logs from multiple log sources, wherein the raw logs at least contain logs corresponding to access requests to the target object; parsing the raw logs to obtain structured logs; generating a MES file queue using the structured logs, wherein one MES file in the MES file queue contains multiple structured logs; and sampling the structured logs contained in each MES file in the MES file queue according to different sampling rates to obtain multiple sampling tables.

5. The method according to claim 4, characterized in that, The step of sampling the structured logs contained in each MES file in the MES file queue according to different sampling rates to obtain multiple sampling tables includes: for each sampling rate, using the task component corresponding to the sampling rate to read the MES files in the MES file queue; assigning each MES file to different task units of the task component; and controlling each task unit to consume the MES files in parallel according to the sampling rate to generate the sampling table.

6. The method according to claim 5, characterized in that, The method of controlling each task unit to consume the MES file in parallel according to the sampling rate to generate the sampling table includes: for each task unit, reading each structured log in the MES file to obtain the business time and analysis object corresponding to the structured log; after determining the business time and analysis object corresponding to the structured log each time, maintaining a request count counter according to the business time and analysis object, with different request count counters used to indicate the number of times different analysis objects are accessed per minute; generating the sampling table according to the request count counter, the sampling rate corresponding to the task component to which the task unit belongs, and the sampling effectiveness count, with the sampling effectiveness count used to indicate the triggering condition for sampling according to the sampling rate.

7. The method according to claim 6, characterized in that, The step of generating the sampling table based on the request count counter, the sampling rate corresponding to the task component to which the task unit belongs, and the sampling effectiveness count includes: after each maintenance of the request count counter, when the count of the request count counter is less than or equal to the sampling effectiveness count, writing the structured log into the sampling table corresponding to the sampling rate; when the count of the request count counter is greater than the sampling effectiveness count, determining whether to write the structured log into the sampling table based on the count of the request count counter and the reciprocal of the sampling rate.

8. The method according to claim 6, characterized in that, Before maintaining the request count counter based on the business time and analysis object corresponding to the structured log each time it is read, the process further includes: determining the number of raw logs generated due to accessing the target object within different minutes based on raw logs collected from multiple log sources; determining a target minute based on the number of raw logs generated due to accessing the target object within different minutes, wherein the target minute has the highest number of raw logs generated due to accessing the target object compared to other minutes within the same timeframe; and starting the request count counter for the target minute, wherein the request counter is used to maintain the number of raw logs generated due to accessing the target object within a valid window, and the valid window is a period of time centered on the target minute.

9. The method according to claim 6, characterized in that, Also includes: Every second time interval, the daily request count of each analyzed object in the total request count cache is updated according to the request count counters of each object. Every second time interval, the independent number corresponding to each aggregation dimension in the independent number cache is updated according to the request count counter.

10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 9.

Images