Log processing methods, log query methods, equipment, media and products

Abstract

This application provides a log processing method, a log query method, a device, a medium, and a product that can be applied to the big data field. The log processing method includes: in response to receiving log data, if the difference between the physical timestamp and logical timestamp of the log data exceeds a preset threshold, determining the log data as out-of-order log data and writing the out-of-order log data into an out-of-order cache pool; if the time interval of the out-of-order log data overlaps with the time interval of log data in the cold storage area, merging the out-of-order log data with the overlapping log data to obtain a log merging result, with the cold storage area used to store sequential log data; and writing the log merging result into the cold storage area as sequential log data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a log processing method, a log query method, a device, a medium, and a product. Background Technology

[0002] Log management is an indispensable part of system monitoring and troubleshooting. Logs not only record the system's operational status but also serve as crucial evidence for troubleshooting and performance optimization. To balance performance, functionality, and storage costs, out-of-order logs are only received and stored within a reasonable timeframe. If the delay is too long, the server will directly refuse to write to them. Adjusting the storage time for out-of-order logs incurs significant memory costs, such as requiring in-memory sorting. Therefore, there is an urgent need for out-of-order data processing and querying methods to effectively reduce storage costs and improve query efficiency. Summary of the Invention

[0003] In view of the above problems, this application provides a log processing method, a log query method, a device, a medium, and a product.

[0004] According to a first aspect of this application, a log processing method is provided, comprising: in response to receiving log data, if the difference between the physical timestamp and the logical timestamp of the log data is greater than a preset threshold, determining the log data as out-of-order log data and writing the out-of-order log data into an out-of-order cache pool; if the time interval of the out-of-order log data overlaps with the time interval of the log data in the cold storage area, merging the out-of-order log data with the overlapping log data to obtain a log merging result, wherein the cold storage area is used to store sequential log data; and writing the log merging result into the cold storage area as sequential log data.

[0005] The second aspect of this application provides a log query method, comprising: in response to receiving a log query request, parsing the log query request to obtain a query time interval and a query label; dividing the log query request into multiple subqueries based on the query time interval and the query label; and executing a query on any one of the multiple subqueries from at least one of a cold storage area, a hot storage area, an out-of-order buffer pool, and an isolated storage area to obtain a target query result.

[0006] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0007] A fourth aspect of this application also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0008] The fifth aspect of this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0009] According to the log processing method provided in this application, log data can be identified as out-of-order log data if the difference between the physical timestamp and logical timestamp of the log data exceeds a preset threshold. If the time intervals of the out-of-order log data overlap with the time intervals of the log data in the cold storage area, the out-of-order log data can be merged with the log data in the overlapping interval to obtain a log merging result. The log merging result is written to the cold storage area as sequential log data, avoiding the resource waste caused by sorting the out-of-order log data in memory, reducing the number of disk input / output operations, and effectively reducing data access time and complexity. Attached Figure Description

[0010] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0011] Figure 1 The illustration shows application scenario diagrams of log processing methods, log query methods, devices, media, and products according to embodiments of this application.

[0012] Figure 2 A flowchart of a log processing method according to an embodiment of this application is shown.

[0013] Figure 3 A flowchart illustrating the merging of log data between the isolated zone and the cold storage zone according to an embodiment of this application is shown.

[0014] Figure 4 A flowchart of a log processing method according to another embodiment of this application is shown.

[0015] Figure 5 A flowchart of a log query method according to an embodiment of this application is shown.

[0016] Figure 6 A flowchart of the subquery execution process according to an embodiment of this application is shown.

[0017] Figure 7 A flowchart of a log query method according to another embodiment of this application is shown.

[0018] Figure 8 A structural block diagram of a log processing apparatus according to an embodiment of this application is shown.

[0019] Figure 9A structural block diagram of a log query apparatus according to an embodiment of this application is shown.

[0020] Figure 10 A block diagram of an electronic device suitable for implementing a log processing method and a log query method according to an embodiment of this application is shown. Detailed Implementation

[0021] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0024] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0025] The ELK technology stack (Elasticsearch, Logstash, Kibana, ELK) is a popular log aggregation solution. ELK provides a high-performance search engine, supports concurrent queries, and offers rich input, filtering, and output plugins. However, this approach relies on full-text indexing, resulting in high storage costs, and query latency increases linearly with data volume, especially in scenarios with massive log volumes where minute-level response times are common.

[0026] To address the aforementioned issues, metadata indexing is typically used instead of full-text search to reduce overhead, making it specifically designed for high efficiency and low cost. A log management system based on metadata indexing may include the following steps: a log collector acquires log information; the master server stores the log information according to a preset line format; and the log platform queries the log information through the application programming interface (API) provided by the master server.

[0027] However, the problem with the above method is that, in order to balance performance, functionality, and storage costs, out-of-order logs are only received and stored within a reasonable timeframe; by default, out-of-order logs within one hour can be written. When the delay exceeds one hour, the server will directly refuse to write. If this timeframe is adjusted, sorting needs to be performed in memory, resulting in significant memory costs.

[0028] Therefore, there is an urgent need for a method for processing and querying out-of-order data to effectively reduce storage costs and improve query efficiency.

[0029] In view of this, embodiments of this application provide a log processing method, including: in response to receiving log data, if the difference between the physical timestamp and logical timestamp of the log data is greater than a preset threshold, determining the log data as out-of-order log data and writing the out-of-order log data into an out-of-order cache pool; if the time interval of the out-of-order log data overlaps with the time interval of the log data in the cold storage area, merging the out-of-order log data with the log data in the overlapping interval to obtain a log merging result, wherein the cold storage area is used to store sequential log data; and writing the log merging result into the cold storage area as sequential log data.

[0030] Figure 1 The illustration shows application scenario diagrams of log processing methods, log query methods, devices, media, and products according to embodiments of this application.

[0031] like Figure 1As shown, the application scenarios of the log processing method, log query method, device, medium, and product in this application embodiment include a query device 101, a first network 102, a computer storage device 103, a second network 104, and a data source 105. The query device 101 can be various electronic devices with a display screen and supporting querying, including but not limited to smartphones, tablets, laptops, and desktop computers. The first network 102 is used as a medium to provide a communication link between the query device 101 and the computer storage device 103. A log query service is deployed in the computer storage device 103, and log queries can be implemented using the log query service. The second network 104 is used as a medium to provide a communication link between the computer storage device 103 and the data source 105. The first network 102 and the second network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables. The data source 105 is a device for storing logs and can be a back-end physical storage medium, such as a hard disk drive, a solid-state drive, or a storage tape. It can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, a cloud server that provides basic cloud computing services, or a node server on a blockchain.

[0032] It should be understood that Figure 1 The number of query devices and data sources shown is merely illustrative. Depending on implementation needs, any number of query devices and data sources can be used.

[0033] The following will be based on Figure 1 The described scene, through Figures 2-7 The log processing method and log query method of the application embodiments are described in detail.

[0034] Figure 2 A flowchart of a log processing method according to an embodiment of this application is shown.

[0035] like Figure 2 As shown, the log processing method of this embodiment includes operations S210 to S230.

[0036] In operation S210, in response to receiving log data, if the difference between the physical timestamp and the logical timestamp of the log data is greater than a preset threshold, the log data is determined to be out-of-order log data and written to the out-of-order cache pool.

[0037] In operation S220, when there is an overlap between the time interval of the out-of-order log data and the time interval of the log data in the cold storage area, the out-of-order log data and the overlapping log data are merged to obtain the log merging result.

[0038] In operation S230, the log merge result is written to the cold storage area as sequential log data.

[0039] In response to received log data, if the difference between the physical timestamp and logical timestamp of the log data exceeds a preset threshold, the log data can be identified as out-of-order log data. Based on the log data's physical timestamp (PTS) and logical timestamp (LTS), the real-time nature of the log data can be determined at the receiving layer. The physical timestamp represents the time the log was generated, and the logical timestamp represents the time the log was sent. For out-of-order data caused by network failures or jitter, the physical timestamp will be earlier than the logical timestamp. Out-of-order log data can be determined by comparing the difference Δt between the physical and logical timestamps with a preset threshold; the difference between the physical and logical timestamps can be obtained by subtracting the logical timestamp from the physical timestamp.

[0040] The embodiments of this application do not limit the preset threshold. The preset threshold is generally selected as half of the maximum time log data can be cached in the hot storage area. For example, after receiving a log write, the maximum time the log data can be cached in the hot storage area is 2 hours. After this timeout, it will be compressed and written to the cold storage area. Because the log data write frequency is not fixed, high write scenarios will reach the disk write condition much less than 2 hours, while low write scenarios may reach 2 hours. Therefore, the average value can be taken for the preset threshold. Furthermore, considering the need to sort and merge sequential and out-of-order logs in the hot storage area, a large window would put significant pressure on memory; therefore, a larger threshold is not set.

[0041] When Δt exceeds a preset threshold, it indicates that the log data latency is too high, making it impossible to sort the data in memory. This is because the larger the time span, the more data is cached in memory, thus increasing the memory cost of sorting. In this case, the log data needs to be classified as out-of-order and written to the out-of-order buffer pool.

[0042] The out-of-order buffer pool is used because out-of-order log data is relatively smaller compared to normal log data. If all incoming log data were written to disk immediately without caching it in memory, it would result in significant disk I / O. Furthermore, it cannot be guaranteed that out-of-order log data from the same time period will not be received again after the initial write. If it is written to disk twice, it would need to be merged into a single data block later, causing unnecessary overhead. Therefore, simple sorting and combining can be performed first in the out-of-order buffer pool.

[0043] Cold storage areas can be used to store infrequently accessed log data and sequential log data. Hot storage areas can be used to store frequently accessed log data.

[0044] When the time intervals of out-of-order log data overlap with the time intervals of log data in the cold storage area, the out-of-order log data can be merged with the log data in the overlapping interval to obtain a log merge result. That is, when merging log data of a certain time interval in the cold storage area, if an overlap is detected between that time interval and the time interval of the out-of-order log data, the log data of that time interval in the cold storage area and the out-of-order log data can be merged, and the log merge result is written to the cold storage area as sequential log data.

[0045] By acquiring log data, if the difference between the physical and logical timestamps of the log data exceeds a preset threshold, the log data can be identified as out-of-order log data. If the time intervals of the out-of-order log data overlap with the time intervals of the log data in the cold storage area, the out-of-order log data can be merged with the overlapping log data to obtain a log merge result. The merged log result is written to the cold storage area as sequential log data, avoiding the resource waste caused by sorting the out-of-order log data in memory, reducing the number of disk I / O operations, and effectively reducing data access time and complexity.

[0046] According to an embodiment of this application, the out-of-order cache pool includes multiple isolated areas; wherein, writing out-of-order log data into the out-of-order cache pool includes: determining a target isolated area among the multiple isolated areas that has the same label as the out-of-order log data and whose time interval of the log data includes the time interval of the out-of-order log data, wherein the label represents the computing device associated with the out-of-order log data; and writing the out-of-order log data into the target isolated area.

[0047] An out-of-order cache pool can include multiple isolated regions. Out-of-order log data is written to the out-of-order cache pool, forming isolated regions based on the time intervals and tags corresponding to the out-of-order log data. Tags can represent computing devices or related process data associated with the out-of-order log data. For example, tags could include "First Server," "Second Server," "First Server First Process," "Second Server First Process," and so on. Isolated regions can represent log data from different time periods with different tags.

[0048] A target isolated region can be identified from multiple isolated regions where the label of the out-of-order log data is the same as that of the out-of-order log data and the time interval of the log data includes the time interval of the out-of-order log data. The out-of-order log data is then written to this target isolated region. For example, there are three isolated regions, with time intervals represented in 24-hour format. The first isolated region corresponds to the time interval from 2 AM to 4 AM and is labeled "Server 1"; the second isolated region corresponds to the time interval from 3 AM to 7 AM and is labeled "Server 2"; the third isolated region corresponds to the time interval from 7 AM to 11 AM and is labeled "Server 1"; and the out-of-order log data corresponds to the time interval from 3 AM to 4 AM and is labeled "Server 1". The target isolated region among these three isolated regions, where the label of the out-of-order log data is the same as that of the out-of-order log data and the time interval of the log data includes the time interval of the out-of-order log data, is identified as the first isolated region. Therefore, the out-of-order log data is written to the first isolated region.

[0049] According to embodiments of this application, when the time intervals of out-of-order log data overlap with the time intervals of log data in the cold storage area, the out-of-order log data is merged with the overlapping log data to obtain a log merging result. For example, sequential log data within the overlapping interval can be obtained from the cold storage area; the sequential log data and the out-of-order log data stored in the target isolated area are then merged to obtain the log merging result.

[0050] Sequential log data for overlapping regions can be retrieved from the cold storage area. Merging this retrieved sequential log data with the out-of-order log data stored in the target isolated area yields the log merge result. It's important to note that the labels for the sequential log data in the cold storage area are the same as those for the out-of-order log data.

[0051] For example, if the out-of-order log data spans from 4 AM to 5 AM, and when merging log data from 2 AM to 10 AM in the cold storage area, an overlap is detected between this time range and the out-of-order log data. Therefore, the log data from 4 AM to 5 AM in the cold storage area and the out-of-order log data can be merged to obtain the merged log result. The log data in the merged result is then sorted according to time order and finally written to the cold storage area as sequential log data.

[0052] According to an embodiment of this application, the log processing method further includes: writing log data in any of the multiple isolated areas to a cold storage area when the number of accesses to any isolated area within a predetermined time period exceeds a preset value; and / or writing log data in any isolated area to a cold storage area when no out-of-order log data is written to any isolated area within a preset time period.

[0053] If any of the multiple isolated zones is accessed more than a preset value within a predetermined time period, the log data in that isolated zone can be written to the cold storage area. In other words, if a user frequently queries the log data in a particular isolated zone, the log data in that isolated zone can be written to the cold storage area.

[0054] If no out-of-order log data is written to any isolated area within a preset time period, the log data in that isolated area will be written to the cold storage area. In other words, if no new out-of-order log data is written to a certain isolated area, the log data in that isolated area can be written to the cold storage area.

[0055] It is important to note that in both of the above scenarios, before writing to the cold storage area, the out-of-order log data in the isolated area and the log data in the cold storage area need to be merged and sorted. Finally, the ordered log data obtained by sorting in chronological order is written to the cold storage area.

[0056] During the log data merging process, it is necessary to access the global time of the log data to confirm the correspondence between the time intervals and tags of the log data in the cold storage area, the isolated storage area, and the out-of-order cache pool.

[0057] Figure 3 A flowchart illustrating the merging of log data between the isolated zone and the cold storage zone according to an embodiment of this application is shown.

[0058] like Figure 3 As shown, if no out-of-order log data is written to any isolated region 302 within a preset time period, the merging engine 301 can be triggered. First, writing out-of-order log data to isolated region 302 can be prevented. Then, log data for the overlapping interval is requested from cold storage region 303, which returns the log data for the overlapping interval. The out-of-order log data in isolated region 302 can be merged with the log data in cold storage region 303 to obtain the log merge result. The log merge result is written to cold storage region 303 as sequential log data. The original log data in cold storage region 303 is then deleted. After successfully writing log data to cold storage region 303, the log data in isolated region 302 can be deleted.

[0059] According to an embodiment of this application, the log processing method further includes: if the available storage space in any isolated area is less than a preset storage threshold, migrating the stored log data in any isolated area to an isolated storage area.

[0060] If the available storage space in any isolated zone is less than a preset storage threshold, the log data stored in any isolated zone can be migrated to an isolated storage zone. Isolated storage zones can be used to store log data that is isolated or not associated with other log data.

[0061] Isolated storage areas and cold storage areas can use the same backend storage, only logically separated. During disk writes, isolated log data is generated based on its state in the out-of-order buffer pool. Isolated log data indicates that the time intervals of these out-of-order log data may be discontinuous, and dividing them into blocks in an isolated format significantly reduces the amount of data accessed during subsequent merging and querying.

[0062] According to an embodiment of this application, the log processing method further includes: when there is an overlap between the time interval of log data in the isolated storage area and the time interval of log data in the cold storage area, obtaining the sequential log data of the overlapping interval from the cold storage area; merging the sequential log data and the log data in the isolated storage area to obtain a log merging result.

[0063] When the time intervals of log data in the isolated storage area overlap with those in the cold storage area, sequential log data for the overlapping intervals can be retrieved from the cold storage area. Merging the sequential log data and the out-of-order log data stored in the isolated storage area yields the log merge result. It's important to note that the labels for the sequential log data in the cold storage area are the same as those for the out-of-order log data in the isolated storage area.

[0064] According to an embodiment of this application, writing the log merging result to the cold storage area in the form of sequential log data includes: sorting multiple log data in the log merging result by time and writing them to the cold storage area.

[0065] The log data in the log merge result is sorted by time and written to the cold storage area as sequential log data to ensure the orderliness of the overall result and improve query efficiency.

[0066] According to an embodiment of this application, the log processing method further includes: writing the log data to a hot storage area when the difference between the physical timestamp and the logical timestamp of the log data is less than or equal to a preset threshold; sorting the log data in the hot storage area according to time to obtain sorted log data; and writing the sorted log data to a cold storage area.

[0067] If the difference between the physical and logical timestamps of log data is less than or equal to a preset threshold, it indicates that the log data latency is within an acceptable range. Log data can be written to the hot storage area as normal data, and sorting can be performed directly within the hot storage area. Finally, it can be written to the cold storage area in the form of data blocks. The hot storage area can include memory, and the cold storage area can include local disks, etc. A data block is a unique set of tags for a specific time period; the same set of tags can contain multiple data blocks for different time periods. Different log data can have different sets of tags, corresponding to different data block groups.

[0068] Figure 4 A flowchart of a log processing method according to another embodiment of this application is shown.

[0069] like Figure 4 As shown, in operation S401, log data is obtained, and operation S402 is executed.

[0070] In operation S402, determine whether the log data is out of order. If the log data is not out of order, execute operation S403. If the log data is out of order, execute operation S404.

[0071] In operation S403, the log data is stored in the hot storage area and sorted, and then operation S407 is executed.

[0072] In operation S404, identify the target isolated region among multiple isolated regions that has the same label as the out-of-order log data and whose time interval of the log data includes the time interval of the out-of-order log data, and then execute operation S405.

[0073] In operation S405, out-of-order log data is written to the target isolated area, and operation S406 is executed.

[0074] In operation S406, if the available storage space in any isolated zone is less than the preset storage threshold, the stored log data in any isolated zone is migrated to the isolated storage zone, and operation S407 is executed.

[0075] In operation S407, the data is stored in the cold storage area, and operation S408 is executed.

[0076] During operation S408, global index update.

[0077] Log data can be retrieved, and then it can be determined whether the log data is out-of-order. If the log data is out-of-order, it is stored in an out-of-order cache pool. If the log data is not out-of-order, it is stored in a hot storage area. The log data in the hot storage area is sorted, and the sorted log data is stored in a cold storage area. A target island area is identified from multiple isolated island areas that has the same label as the out-of-order log data and whose time interval includes the time interval of the out-of-order log data. The out-of-order log data is written to the target island area. If the available storage space in any island area is less than a preset storage threshold, the stored log data in any island area can be migrated to the island storage area. Based on a trigger mechanism, the log data in the data islands can be merged with the log data in the cold storage area to obtain a log merge result. The log merge result is written to the cold storage area as sequential log data. Triggering mechanisms can include overlapping time intervals between log data in data silos and log data in cold storage areas; any silo being accessed more than a preset number of times within a predetermined time period; and no out-of-order log data being written to any silo within a preset time period. The log processing method can dynamically maintain a global time map to support global index updates. The global time map includes the tags, silo start time, silo end time, and storage location for all log data in the out-of-order buffer pool and silo storage areas. Storage locations may include, for example, within the buffer pool or the silo storage area.

[0078] By maintaining a global time map and merging and rearranging log data based on time intervals, the problem of out-of-order data writing and management can be solved. By managing the out-of-order buffer pool and isolated storage areas, out-of-order data is stored in isolation, and the orderliness and integrity of merging and querying are guaranteed based on a dynamic global time map.

[0079] Figure 5 A flowchart of a log query method according to an embodiment of this application is shown.

[0080] like Figure 5 As shown, the log query method in this embodiment includes operations S510 to S530.

[0081] When operating S510, in response to receiving a log query request, the log query request is parsed to obtain the query time range and query label.

[0082] When operating S520, log query requests are divided into multiple subqueries based on the query time range and query label.

[0083] In operation S530, for any one of the multiple subqueries, the query is executed from at least one of the cold storage area, hot storage area, out-of-order buffer pool, and islanded storage area to obtain the target query result.

[0084] In response to a received log query request, the request can be parsed to obtain the query time range and query tags. The query time range may include a start timestamp and an end timestamp. The query tags may include data type, data source, query priority, etc. However, this is not limited to these aspects, and the embodiments of this application do not impose such limitations.

[0085] Based on the query time range and query tags, log query requests can be divided into multiple subqueries. For any of these subqueries, the query can be executed from at least one of the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area. The retrieved log data is then sorted by time to obtain ordered target query results, ensuring the overall orderliness of the results.

[0086] By responding to received log query requests and parsing them, the query time range and query tags can be obtained, thereby accurately locating the time range and tags of the target log data and reducing interference from irrelevant data. Based on the query time range and query tags, the log query request can be divided into multiple subqueries, which can be executed in parallel, making full use of system resources and shortening query time. For any of the multiple subqueries, the query can be executed from at least one of the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area to obtain the target query result, ensuring the completeness and accuracy of the query results.

[0087] According to an embodiment of this application, a log query request is divided into multiple subqueries based on a query time interval and query tags, including: dividing the query time interval into multiple subquery time intervals based on a time interval division strategy, wherein the time interval division strategy includes at least one of division according to a fixed time length, division according to data volume distribution, and division according to time granularity; dividing the log query request into multiple tag results based on query tags, wherein each tag result corresponds to one or more tags; and generating multiple subqueries based on the subquery time intervals and tag results.

[0088] A time interval partitioning strategy can divide a query time interval into multiple subquery time intervals. This strategy can include at least one of the following: partitioning by fixed time length, partitioning by data volume distribution, and partitioning by time granularity. Partitioning by fixed time length could mean subquery time intervals are defined as hourly or daily intervals. Partitioning by data volume distribution can divide the time interval into subquery time intervals with similar data volumes based on the data distribution. Partitioning by time granularity allows you to select appropriate time precision based on the query's accuracy requirements, such as seconds, minutes, or hours.

[0089] Log query requests can be divided into multiple tagged results based on query tags. Each tagged result can correspond to one or more tags. For example, query tags can include data type, data source, etc. Data type can include "error log," "access log," etc. Data source can include "server A," "server B," etc., but is not limited to these; the embodiments of this application do not limit the query tags. Each tagged result can correspond to one or more tags. For example, the tagged result is "error log, server A." Multiple subqueries can be generated based on the subquery time interval and the tagged results. For example, if the subquery time interval is from 2 AM to 3 AM, the tagged result is "error log, server A."

[0090] According to an embodiment of this application, a query is executed from at least one of a cold storage area, a hot storage area, an out-of-order buffer pool, and an isolated storage area to obtain a target query result, including: determining at least one of the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area as the target query area based on the subquery time interval and subquery label of any subquery; and executing any subquery for the target query area to obtain the target query result.

[0091] Based on the subquery time interval and subquery label of any subquery, at least one of the following can be identified as the target query region: cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area. Executing any subquery against the target query region yields the target query result.

[0092] For example, if the subquery's time interval is from 2 AM to 3 AM, and the tag results are "error log, server A", we can perform time-based matching on the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area to determine if there are corresponding time intervals in these areas. Then, based on the tag results, we can match them with the cold storage area, hot storage area, and out-of-order buffer pool respectively, determining that there are corresponding tag results in the cold storage area. Therefore, the cold storage area can be identified as the target query region.

[0093] According to an embodiment of this application, executing any subquery for a target query region to obtain the target query result includes: executing any subquery for the target query region to obtain the subquery result; and sorting the multiple subquery results in chronological order to obtain the target query result.

[0094] You can execute subqueries on the target query area to get the subquery results. You can sort multiple subquery results in chronological order to get the target query result.

[0095] For example, the target query region for the first subquery is the cold storage area, and the target query region for the second subquery is both the cold storage area and the out-of-order buffer pool. The first subquery can be executed to obtain its result, and the second subquery can be executed to obtain its result and the third subquery's result. The results of the first, second, and third subqueries are then sorted in chronological order to obtain the target query result.

[0096] Figure 6 A flowchart of the subquery execution process according to an embodiment of this application is shown.

[0097] like Figure 6 As shown, log query requests can be divided into multiple subqueries based on the query time interval and query label. Each subquery has a subquery time interval and a subquery label. For example, the subquery label corresponding to the first subquery is server A, and the subquery time interval corresponding to the first subquery is T1. Based on the subquery time interval and subquery label of the first subquery, matching can be performed from the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area, thereby determining the cold storage area as the target query area. Executing the first subquery against the target query area yields the first log.

[0098] The subquery tag for the second subquery is server B, and the subquery time interval is T2. Based on the subquery time interval and subquery tag, matching can be performed from the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area, respectively. This determines the isolated storage area as the target query region. Executing the second subquery against the target query region yields the second log.

[0099] The subquery label for the third subquery is server A, and the subquery time interval is T3. Based on the subquery time interval and subquery label, matching can be performed from the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area, thus determining the cold storage area as the target query area. Executing the third subquery against the target query area yields the third log.

[0100] The subquery label corresponding to the (n-1)th subquery is server A, and the subquery time interval is Tn-1. Based on the subquery time interval and subquery label of the (n-1)th subquery, matching can be performed from the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area, respectively. This determines the cold storage area as the target query area. Executing the (n-1)th subquery against the target query area yields the (n-1)th log entry.

[0101] The subquery label corresponding to the nth subquery is server C, and the subquery time interval corresponding to the nth subquery is Tn. Based on the subquery time interval and subquery label of the nth subquery, matching can be performed from the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area, respectively. This determines the hot storage area as the target query area. Executing the nth subquery against the target query area yields the nth log; sorting the logs from the first to the nth log in chronological order provides the target query result.

[0102] Figure 7 A flowchart of a log query method according to another embodiment of this application is shown.

[0103] like Figure 7 As shown, upon receiving a log query request 701, the query parser 702 can parse the log query request to obtain the query time range and query label, and then send it to the routing decision engine 703. After receiving the query time range and query label, the time-based routing decision engine 703 can divide the log query request into multiple subqueries with subquery time ranges and subquery labels. The cold storage scanner 704 can scan the cold storage area, the hot storage scanner 705 can scan the hot storage area, the out-of-order buffer pool scanner 706 can scan the out-of-order buffer pool, and the isolated storage area scanner 707 can scan the isolated storage area, ultimately determining the target query area. Then, based on executing subqueries from the target query area, the subqueries can be executed in chronological order according to the time axis, and the query results can be sent to the streaming merge engine 708. The streaming merge engine 708 can guarantee the ordered nature of multiple subquery results within the same time period. When both sequential and out-of-order logs exist within a certain time period, the sequential and out-of-order logs are reordered in memory to obtain the target query result 709. This ensures the overall orderliness of the query results. Based on the timeline, routing decisions are made for subqueries to guarantee the overall orderliness of the results and query performance. Ultimately, this solves the problem of servers being unable to store out-of-order data.

[0104] Based on the above log processing method, this application also provides a log processing apparatus. The following will be combined with... Figure 8 The device is described in detail.

[0105] Figure 8 A structural block diagram of a log processing apparatus according to an embodiment of this application is shown.

[0106] like Figure 8 As shown, the log processing device 800 of this embodiment includes an out-of-order log determination module 810, a merging module 820, and an ordered log writing module 830.

[0107] The out-of-order log determination module 810 is used to determine the log data as out-of-order log data in response to receiving log data, if the difference between the physical timestamp and logical timestamp of the log data is greater than a preset threshold, and write the out-of-order log data into the out-of-order cache pool.

[0108] The merging module 820 is used to merge out-of-order log data with overlapping log data when there is an overlap between the time interval of out-of-order log data and the time interval of log data in the cold storage area, to obtain the log merging result. The cold storage area is used to store sequential log data.

[0109] The sequential log writing module 830 is used to write the log merging results to the cold storage area in the form of sequential log data.

[0110] The out-of-order cache pool comprises multiple isolated regions. The out-of-order log determination module 810 includes a target isolated region determination unit and a target isolated region writing unit. The target isolated region determination unit is used to determine the target isolated region among the multiple isolated regions that has the same label as the out-of-order log data and whose time interval includes the time interval of the out-of-order log data. The label indicates the computing device associated with the out-of-order log data. The target isolated region writing unit is used to write the out-of-order log data into the target isolated region.

[0111] The merging module 820 includes an overlapping data acquisition unit and a merging unit. The overlapping data acquisition unit is used to acquire sequential log data of overlapping intervals from the cold storage area; the merging unit is used to merge the sequential log data and the out-of-order log data stored in the target isolated area to obtain the log merging result.

[0112] The log processing device 800 in this embodiment further includes: a first writing module and a second writing module. The first writing module is used to write log data in any isolated area to a cold storage area when the number of accesses to any isolated area among a plurality of isolated areas exceeds a preset value within a predetermined time period; the second writing module is used to write log data in any isolated area to a cold storage area when no out-of-order log data is written to any isolated area within the preset time period.

[0113] The log processing apparatus 800 in this embodiment further includes a migration module. The migration module is used to migrate stored log data from any isolated storage area to an isolated storage area when the available storage space in any isolated storage area is less than a preset storage threshold.

[0114] The log processing apparatus 800 in this embodiment further includes an overlapping log acquisition module and a log merging module. The overlapping log acquisition module is used to acquire sequential log data of the overlapping time interval from the cold storage area when there is an overlap between the time interval of log data in the isolated storage area and the time interval of log data in the cold storage area; the log merging module is used to merge the sequential log data and the log data in the isolated storage area to obtain a log merging result.

[0115] The sequential log writing module 830 includes a time sorting unit. This time sorting unit is used to sort multiple log data entries from the log merging result by time and write them to the cold storage area.

[0116] The log processing device 800 in this embodiment further includes a third writing module, a sorting module, and a fourth writing module. The third writing module is used to write log data to a hot storage area when the difference between the physical timestamp and the logical timestamp of the log data is less than or equal to a preset threshold; the sorting module is used to sort the log data in the hot storage area according to time to obtain sorted log data; and the fourth writing module is used to write the sorted log data to a cold storage area.

[0117] Based on the above log query method, this application also provides a log query device. The following will combine... Figure 9 The device is described in detail.

[0118] Figure 9 A structural block diagram of a log query apparatus according to an embodiment of this application is shown.

[0119] like Figure 9 As shown, the log query device 900 of this embodiment includes a parsing module 910, a subquery partitioning module 920, and an execution module 930.

[0120] The parsing module 910 is used to respond to a received log query request by parsing the log query request and obtaining the query time range and query tag.

[0121] The subquery partitioning module 920 is used to divide log query requests into multiple subqueries based on the query time range and query tags.

[0122] The execution module 930 is used to execute the query from at least one of the cold storage area, hot storage area, out-of-order buffer pool and islanded storage area for any of the multiple subqueries, and obtain the target query result.

[0123] The subquery partitioning module 920 includes: a first partitioning unit, a second partitioning unit, and a subquery generation unit. The first partitioning unit is used to divide the query time interval into multiple subquery time intervals based on a time interval partitioning strategy, which includes at least one of partitioning by fixed time length, partitioning by data volume distribution, and partitioning by time granularity. The second partitioning unit is used to partition the log query request into multiple tag results based on query tags, with each tag result corresponding to one or more tags. The subquery generation unit is used to generate multiple subqueries based on the subquery time intervals and tag results.

[0124] The execution module 930 includes a target query region determination unit and an execution unit. The target query region determination unit is used to determine at least one of the following as the target query region based on the subquery time interval and subquery label of any subquery: cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area. The execution unit is used to execute any subquery for the target query region to obtain the target query result.

[0125] The execution unit includes an execution subunit and a sorting subunit. The execution subunit is used to execute any subquery for the target query range and obtain the subquery result; the sorting subunit is used to sort the results of multiple subqueries in chronological order and obtain the target query result.

[0126] Any multiple modules among the out-of-order log determination module 810, merging module 820, sequential log writing module 830, parsing module 910, subquery partitioning module 920, and execution module 930 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. At least one of the out-of-order log determination module 810, merging module 820, sequential log writing module 830, parsing module 910, subquery partitioning module 920, and execution module 930 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the out-of-order log determination module 810, merging module 820, sequential log writing module 830, parsing module 910, subquery partitioning module 920, and execution module 930 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0127] Figure 10 A block diagram of an electronic device suitable for implementing a log processing method and a log query method according to an embodiment of this application is shown.

[0128] like Figure 10 As shown, an electronic device 1000 according to an embodiment of this application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0129] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 1002 and / or RAM 1003. It should be noted that programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0130] Electronic device 1000 may also include an input / output (I / O) interface 1005, which is also connected to bus 1004. Electronic device 1000 may also include one or more of the following components connected to input / output (I / O) interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 1010 as needed so that computer programs read from it can be installed into storage section 1008 as needed.

[0131] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the log processing method and log query method according to the embodiments of this application.

[0132] Computer-readable storage media can be non-volatile computer-readable storage media, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0133] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the log processing method and log query method provided in the embodiments of this application.

[0134] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this application embodiment. The systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0135] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0136] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, it performs the functions defined in the system of this application embodiment. The systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0137] Program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0140] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A log processing method, characterized in that, The method includes: In response to receiving log data, if the difference between the physical timestamp and the logical timestamp of the log data is greater than a preset threshold, the log data is determined to be out-of-order log data, and the out-of-order log data is written into an out-of-order cache pool, the out-of-order cache pool including multiple isolated areas, and the preset threshold is set to half of the preset duration for which the log data is cached in the hot storage area; If the time intervals of the out-of-order log data and the time intervals of the log data in the cold storage area overlap, the out-of-order log data and the overlapping log data are merged to obtain a log merging result. The cold storage area is used to store sequential log data. The log merge result is written to the cold storage area as sequential log data; and / or If any of the multiple isolated areas is accessed more than a preset value within a predetermined time period, the log data in that isolated area will be written to the cold storage area. and / or If no out-of-order log data is written to any of the isolated islands within a preset time period, the log data in any of the isolated islands is written to the cold storage area. Writing the out-of-order log data to the out-of-order cache pool includes: Identify a target isolated region among the plurality of isolated regions that has the same label as the out-of-order log data and whose time interval of the log data includes the time interval of the out-of-order log data, wherein the label represents the computing device associated with the out-of-order log data; Write the out-of-order log data into the target isolated area.

2. The method according to claim 1, characterized in that, When the time intervals of the out-of-order log data and the log data in the cold storage area overlap, the out-of-order log data and the overlapping log data are merged to obtain a log merging result, including: Obtain sequential log data for the overlapping time intervals from the cold storage area; The sequential log data and the out-of-order log data stored in the target isolated area are merged to obtain the log merging result.

3. The method according to claim 1, characterized in that, The method further includes: If the available storage space in any of the isolated zones is less than a preset storage threshold, the stored log data in any of the isolated zones will be migrated to the isolated storage zone.

4. The method according to claim 3, characterized in that, The method further includes: If the time intervals of the log data in the isolated storage area overlap with the time intervals of the log data in the cold storage area, the sequential log data of the overlapping interval is obtained from the cold storage area. The sequential log data and the log data in the isolated storage area are merged to obtain the log merging result.

5. The method according to claim 1, characterized in that, The step of writing the log merging result to the cold storage area in the form of sequential log data includes: The log data from the log merging result are sorted by time and written to the cold storage area.

6. The method according to claim 1, characterized in that, The method further includes: If the difference between the physical timestamp and the logical timestamp of the log data is less than or equal to the preset threshold, the log data will be written to the hot storage area. The log data in the hot storage area is sorted by time to obtain sorted log data; and Write the sorting log data to the cold storage area.

7. A log query method, characterized in that, The method includes: In response to receiving a log query request, the log query request is parsed to obtain the query time range and query tag; Based on the query time range and the query tag, the log query request is divided into multiple sub-queries; For any one of the multiple subqueries, the query is executed from at least one of the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area to obtain the target query result, wherein the log data in the cold storage area, the hot storage area, the out-of-order buffer pool, and the isolated storage area are obtained by the method according to any one of claims 1 to 6.

8. The method according to claim 7, characterized in that, The log query request is divided into multiple sub-queries based on the query time interval and the query tag, including: The query time interval is divided into multiple sub-query time intervals based on a time interval partitioning strategy. The time interval partitioning strategy includes at least one of partitioning according to a fixed time length, partitioning according to data volume distribution, and partitioning according to time granularity. Based on the query tags, the log query request is divided into multiple tag results, and each tag result corresponds to one or more tags; Multiple subqueries are generated based on the subquery time interval and the label results.

9. The method according to claim 7, characterized in that, The step of executing a query from at least one of the cold storage area, hot storage area, out-of-order buffer pool, and isolated storage area to obtain the target query result includes: Based on the subquery time interval and subquery label of any subquery, at least one of the cold storage area, the hot storage area, the out-of-order buffer pool, and the isolated storage area is determined as the target query area; Execute any of the subqueries for the target query region to obtain the target query result.

10. The method according to claim 8, characterized in that, The step of executing any subquery on the target query region to obtain the target query result includes: Execute any of the subqueries for the target query region to obtain the subquery results; The results of the multiple subqueries are sorted according to time order to obtain the target query result.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images