Log collection method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a log collection method, which can be applied to the field of financial technology or other technical fields. The log collection method comprises: obtaining a to-be-collected data set, wherein the to-be-collected data set comprises log data of at least one log type; for each log type, obtaining a first data amount of the log type from a flow restrictor, wherein the first data amount comprises a data amount of log data of the log type that has been collected by the flow restrictor currently; when the first data amount of any one log type meets a first preset condition, eliminating the log data of the log type from the to-be-collected data set to obtain a target data set; collecting the log data in the target data set and sending the log data to a cache channel. The present disclosure also provides a log collection device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and specifically to a log collection method, apparatus, device, storage medium, and program product. Background Technology

[0002] Flume is a log collection system that enables highly available, highly reliable, and distributed massive log collection, aggregation, and transmission. Flume supports customizing various data senders within the log system for data collection; at the same time, Flume provides the ability to perform simple data processing and write to various data receivers.

[0003] Traditional Flume consists of a source, a channel, and a sink. The source receives data, encapsulates it into events, and sends it to the channel. The channel acts as a queue, serving as a buffer between the source and the sink. The sink continuously polls the events in the channel and writes them to storage, an indexing system, or another Flume instance.

[0004] Currently, data filtering can be performed using interceptors. However, the filtering performance of interceptors is limited, and for example, the filtering granularity is difficult to meet the increasingly diverse needs. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a log collection method, apparatus, device, storage medium and program product.

[0006] According to the first aspect of this disclosure, a log collection method is provided, applied at the collection source end, including:

[0007] Obtain the dataset to be collected, wherein the dataset includes log data of at least one log type;

[0008] For each log type, a first data volume for that log type is obtained from the rate limiter, the first data volume including: the amount of log data for that log type that the rate limiter has currently collected;

[0009] When the first data volume of any log type meets the first preset condition, the log data of that log type is removed from the dataset to be collected to obtain the target dataset;

[0010] Collect log data from the target dataset and send it to the cache channel.

[0011] According to embodiments of this disclosure, the collection of log data from the target dataset includes:

[0012] The first data volume of at least one log type in the flow limiter is updated based on the size of each collected log data and the log type of each log data.

[0013] According to embodiments of this disclosure, removing log data of this log type from the dataset to be collected includes:

[0014] Get the first time of this log type in the suppressor, the first time including the time when the last over-limit log information was sent for this log type;

[0015] When the interval between the first time and the current time meets the second preset condition, an over-limit log message is sent according to the log type.

[0016] According to embodiments of this disclosure, the suppressor records multiple log types and a first time for each log type; the log collection method further includes:

[0017] For each log type in the suppressor, the first time of that log type is periodically reset according to a first time interval.

[0018] According to embodiments of this disclosure, the flow limiter records multiple log types and a first data volume for each log type; the log collection method further includes:

[0019] For each log type in the rate limiter, the first data volume of that log type is periodically reset at a second time interval;

[0020] The duration of the first time interval is the same as the duration of the second time interval.

[0021] According to embodiments of this disclosure, obtaining the dataset to be collected includes:

[0022] When the log directory to be collected is refreshed, the log data to be collected is obtained according to the directory to be collected, so as to form the dataset to be collected;

[0023] Wherein, the duration of the first time interval and the duration of the second time interval are both greater than the refresh interval of the log directory to be collected.

[0024] According to embodiments of this disclosure, the log collection method further includes:

[0025] Remove invalid log data from the target dataset. The invalid log data includes log data that has not been collected after a preset time period.

[0026] Remove the log type corresponding to the invalid log data from the rate limiter; and / or, remove the log type corresponding to the invalid log data from the suppressor;

[0027] Wherein, the duration of both the first time interval and the second time interval is less than the preset duration.

[0028] According to embodiments of this disclosure, the rate limiter includes multiple rate limiting components, each rate limiting component recording at least one log type of the first data volume, and different rate limiting components recording different log types.

[0029] A second aspect of this disclosure provides a log collection device applied at a collection source, comprising:

[0030] The first acquisition module is used to acquire a dataset to be collected, wherein the dataset to be collected includes log data of at least one log type;

[0031] The second acquisition module is used to acquire a first data volume of each log type from the rate limiter for each log type, wherein the first data volume includes the amount of log data of that log type that the rate limiter has currently collected;

[0032] The processing module is used to remove log data of that log type from the dataset to be collected when the first data volume of any log type meets the first preset condition, so as to obtain the target dataset;

[0033] The acquisition module is used to collect log data from the target dataset and send it to the cache channel.

[0034] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the log collection method described above.

[0035] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the log collection method described above.

[0036] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the log collection method described above.

[0037] The above one or more embodiments have the following advantages or beneficial effects:

[0038] Rate limiting is implemented based on log type, ensuring that the collection of log data of different types is independent and does not affect each other. Furthermore, because this disclosure filters data before encapsulating it into transactions, it also mitigates the problem of wasted resources at the data collection source. Attached Figure Description

[0039] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0040] Figure 1 The diagram illustrates the module block diagram of an example log collection tool;

[0041] Figure 2 The illustration schematically depicts an application scenario of a log collection method, apparatus, device, storage medium, and program product according to embodiments of the present disclosure.

[0042] Figure 3 One of the flowcharts of a log collection method according to an embodiment of the present disclosure is illustrated schematically;

[0043] Figure 4 A second flowchart illustrating a log collection method according to an embodiment of the present disclosure is shown schematically.

[0044] Figure 5 A flowchart illustrating the sending of over-limit log information according to an embodiment of this disclosure is shown schematically;

[0045] Figure 6 A schematic block diagram of a log collection apparatus according to an embodiment of the present disclosure is shown.

[0046] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a log collection method according to an embodiment of the present disclosure. Detailed Implementation

[0047] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0048] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0049] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0050] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0051] The log collection method, apparatus, device, storage medium, and program product disclosed herein can be used in information transmission services in the financial sector, and can also be used in any sector other than the financial sector. The application areas of the log collection method, apparatus, device, storage medium, and program product disclosed herein are not limited.

[0052] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0053] The following section uses the APM log collection tool as an example to illustrate the traditional log collection process.

[0054] APM is a log message collection tool developed based on Apache Flume. Figure 1 This diagram illustrates the module block diagram of an example log collection tool, such as... Figure 1As shown, the log collection tool includes a collection source 1, a cache channel 2, and a consumer 3. In this example, log data can be filtered using Flume's built-in interceptor 4, which is positioned between the collection source 1 and the cache channel 2. The collection source 1 collects the log data to be collected, and after steps such as information supplementation, encapsulates it into a transaction (event). The interceptor 4 filters the transactions output by the collection source 1 according to a pre-configured strategy and then sends the filtered transactions to the cache channel 2. The consumer 3 retrieves the transactions from the cache channel 2 and sends them to the corresponding receiving end, such as a distributed file system (Hadoop Distributed File System, HDFS), a Hadoop-based data warehouse tool (Hive), or a distributed, column-oriented open-source database (HBase).

[0055] The inventors discovered the following problems with current interceptors during their research:

[0056] 1. Interceptor 4 is bound to the data acquisition source 1. Interceptor 4 can only limit the total amount of transactions output by the data acquisition source 1, and its filtering fineness is limited. For example, when a system exception occurs, the logs about the exception output by the data acquisition source 1 increase, which will consume the limit of interceptor 4. Furthermore, since the limit of interceptor 4 is generally relatively large, if the surge in logs about exceptions in a short period of time does not reach the limit, it will not be subject to rate limiting. A large number of logs about exceptions may cause the receiving end to become overloaded and unstable.

[0057] 2. When using interceptor 4 for rate limiting, the interceptor 4 will only filter the data after the source 1 generates a transaction and transmits it to the interceptor 4. This results in a waste of resources for the source 1.

[0058] In view of the above, embodiments of this disclosure provide a log collection method applied to a collection source, comprising: acquiring a dataset to be collected, the dataset including log data of at least one log type; for each log type, acquiring a first data volume of that log type from a rate limiter, the first data volume including the amount of log data of that log type currently collected by the rate limiter; when the first data volume of any log type meets a first preset condition, removing the log data of that log type from the dataset to be collected to obtain a target dataset; collecting log data from the target dataset and sending it to a cache channel.

[0059] The log collection method of this disclosure can limit the flow of log data based on its log type, thereby ensuring that the collection of log data of different types is independent and does not affect each other, thus achieving fine-grained management of data filtering. Furthermore, since the log collection method of this disclosure can filter data before encapsulating it into transactions, it also improves the problem of wasted resources at the data collection source.

[0060] Figure 2 This illustration schematically depicts an application scenario of a log collection method, apparatus, device, storage medium, and program product according to embodiments of the present disclosure, such as... Figure 2 As shown, application scenario 200 according to this embodiment may include terminal devices 201, 202, and 203, a network 204, and a server 205. Network 204 serves as a medium for providing a communication link between terminal devices 201, 202, and 203 and server 205. Network 204 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0061] Users can use terminal devices 201, 202, and 203 to interact with server 205 via network 204 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 201, 202, and 203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0062] Terminal devices 201, 202, and 203 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0063] Server 205 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 201, 202, and 203 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0064] It should be noted that the log collection provided in this embodiment can generally be performed by server 205. Correspondingly, the log collection device provided in this embodiment can generally be located in server 205. The log collection method provided in this embodiment can also be performed by a server or server cluster that is different from server 205 and capable of communicating with terminal devices 201, 202, 203 and / or server 205. Correspondingly, the log collection device provided in this embodiment can also be located in a server or server cluster that is different from server 205 and capable of communicating with terminal devices 201, 202, 203 and / or server 205.

[0065] It should be understood that Figure 2 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0066] The following will be based on Figure 2 The described scene, through Figures 3-6 The log collection method of the disclosed embodiments is described in detail.

[0067] The embodiments of this disclosure provide a log collection method applied to a collection source. For example, the collection source can refer to a process, application, device, or system. Figure 3 One of the flowcharts of a log collection method according to an embodiment of the present disclosure is illustrated schematically. Figure 4 A second flowchart illustrating a log collection method according to an embodiment of this disclosure is shown schematically, in conjunction with... Figure 3 and Figure 4 As shown, the log collection method in this embodiment includes steps S310 to S340. It should be noted that, although... Figure 3 The steps in the diagram are shown sequentially as indicated by the arrows; however, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0068] In step S310, a dataset to be collected is obtained, which includes log data of at least one log type.

[0069] Optionally, in embodiments of this disclosure, the collection dataset can be obtained through a collection configuration file. For example, the collection configuration file can be scanned, and the log data to be collected can be obtained according to the collection directory configured in the collection configuration file, thereby forming the collection dataset. The method of scanning the collection configuration file can be determined according to actual needs. For example, a scanning cycle can be set, and the collection configuration file can be scanned according to the scanning cycle.

[0070] Optionally, by scanning the collection directory configured in the collection configuration file, the path information of the data to be collected can be obtained, and then the log data to be collected can be obtained based on the path information.

[0071] Optionally, a collection configuration file can correspond to at least one log type. Different collection configuration files correspond to different log types. For example, a one-to-one correspondence between collection configuration files and log types means that by scanning a collection configuration file, log data of a certain log type can be obtained.

[0072] Optionally, multiple log data types can be retrieved in parallel using multiple threads. For example, log data A1 of log type B1 and log data A2 of log type B2 can be extracted simultaneously. Of course, in other specific embodiments, log data A1 and log data A2 can also be extracted in a time-sharing manner, which can be determined according to actual needs and is not limited here.

[0073] In step S320, for each log type, a first data volume for that log type is obtained from the rate limiter. The first data volume includes the amount of log data for that log type that the rate limiter has currently collected.

[0074] In step S330, it is determined whether the first data volume of any log type meets the first preset condition; when the first data volume of any log type meets the first preset condition, the log data of that log type is removed from the dataset to be collected; when the first data volume of a log type does not meet the first preset condition, the log data of that log type is retained to obtain the target dataset.

[0075] In the embodiments of this disclosure, the first preset condition may include a first data volume greater than or equal to a first preset threshold. That is, when the first data volume of any log type is greater than or equal to the first preset threshold, log data of that log type is removed from the dataset to be collected.

[0076] Optionally, a first preset threshold can be configured for each log type in the data collection configuration file. Each log type corresponds to at least one first preset threshold. The first preset thresholds for different log types can be the same or different, depending on actual needs, and are not restricted here. When scanning the data collection configuration file, at least one log type and the first preset threshold corresponding to that log type are extracted from it, and both are written to the rate limiter.

[0077] The following example illustrates the process of obtaining the target dataset, using a dataset consisting of log data A1 of log type B1 and log data A2 of log type B2 as an example.

[0078] In step S320, the first data volume of log type B1 and the first data volume of log type B2 are obtained from the rate limiter. That is, the amount of log data of log type B1 and the amount of log data of log type B2 collected by the rate limiter during the period between the current time and a certain time before the current time. When the first data volume of log type B1 is less than a first preset threshold and the first data volume of log type B2 is greater than the first preset threshold, the log corresponding to log type B2 is removed from the dataset to be collected. That is, log data A2 is removed from the dataset to be collected, and only log data A1 is retained, thereby obtaining the target dataset.

[0079] It should be noted that the first preset threshold corresponding to log type B1 and the first preset threshold corresponding to log type B2 can be the same or different, depending on the actual needs, and no restrictions are imposed here. In step S340, log data in the target dataset is collected and sent to the cache channel.

[0080] In this embodiment of the disclosure, after collecting log data from the target dataset, operations such as supplementing information as described above can be performed. Afterward, the collected log data is encapsulated into a transaction and sent to the cache channel.

[0081] Optionally, after receiving a transaction sent by the data acquisition source, the processor of the cache channel can first send the received transaction to the interceptor for filtering, such as data cleaning; after filtering, the interceptor returns the filtered data to the processor of the cache channel for further processing.

[0082] Therefore, the log collection method of the embodiments of this disclosure can limit the flow according to the log type, so that the collection of log data of different log types is independent and does not affect each other.

[0083] For example, as mentioned above, when the first data volume of log type B2 (e.g., log data corresponding to system anomalies) exceeds a first preset threshold, and the first data volume of log type B1 (regular log data) is less than the first preset threshold, the collection of log data of log type B2 can be restricted, while the collection of log data of log type B1 can remain unrestricted. That is, log data A2 is removed from the dataset to be collected, while log data A1 is retained. In this way, during subsequent collection processes, only log data of log type B1, i.e., log data A1, will be collected from the target dataset. Compared to traditional log collection methods that can only limit the total amount of transaction data output from the collection source, the log collection method of this disclosure, even if the total amount of transaction data output from the collection source does not exceed the limit, can restrict the collection of log data of a certain log type once the data volume of a certain log type exceeds the limit, thereby achieving fine-grained management of data filtering.

[0084] Furthermore, since the log collection method of the embodiments of this disclosure can filter data before encapsulating it into transactions, the log collection method of the embodiments of this disclosure also improves the problem of wasted resources at the collection source.

[0085] The following is combined with Figures 3 to 6 The log collection method of this disclosure will be further described.

[0086] In some specific embodiments, step S340 includes step S341.

[0087] In step S341, the first data volume of at least one log type in the flow limiter is updated according to the size of each collected log data and the log type of each log data.

[0088] In this embodiment of the disclosure, updating the first data volume of the log type in the rate limiter can be, for example, by combining the size of the collected log data with the first data volume to obtain a new first data volume, and using this new first data volume as the updated first data volume. For example, the first data volume may include the number of bytes, and combining the size of the collected log data with the first data volume may include adding the number of bytes of the collected log data to the first data volume.

[0089] In some specific embodiments, the rate limiter includes multiple rate limiting components, such as counters. Each rate limiting component records a first data volume of at least one log type, and different rate limiting components record different log types. For example, each rate limiting component records a first data volume of one log type, that is, a one-to-one correspondence is set between rate limiting components and log types.

[0090] For example, as described above, in step S320, the first data volume of log type B1 is obtained from the rate limiter, which is 10KB in size. That is, during the period between the current time and a certain time before the current time, the data volume of log data of log type B1 collected by the rate limiter is 10KB. Assuming the first preset threshold corresponding to log type B1 is 15KB, since the first data volume of log type B1 is less than the first preset threshold, log data A1 in the dataset to be collected is retained. In step S340, log data A1 is collected and its size is calculated to be 4KB. In step S351, the number of bytes in log data A1 is added to the first data volume of log type B1 to obtain the updated first data volume, which is 14KB.

[0091] It should be noted that in the embodiments of this disclosure, in step S351, when the updated first data volume exceeds the first preset threshold, no restriction operation is taken. Instead, when a new dataset to be collected is obtained again, in step S340, it is determined whether the first data volume exceeds the first preset threshold, and then corresponding operations are taken.

[0092] For example, in the Nth collection process, the dataset to be collected obtained through step S310 is X1, which includes log data A1 of log type B1. In step S320, the first data volume of log type B1 is obtained from the rate limiter, with a size of 10KB. The first preset threshold corresponding to log type B1 is 15KB, and the size of log data A1 is 6KB. Since the first data volume of log type B1 is less than the first preset threshold, the log data A1 of the dataset to be collected X1 is retained, thereby obtaining the target dataset. In step S340, log data A1 is collected from the target dataset, wherein the updated first data volume is 16KB after superimposing the number of bytes of log data A1 with the first data volume of log type B1 in step S351. Here, N is a positive integer.

[0093] During the (N+1)th data collection process, the dataset to be collected, obtained through step S310, is X2. This dataset includes log data A3 of log type B1. In step S320, the first data volume of log type B1 is obtained from the rate limiter, with a size of 16KB. The first preset threshold corresponding to log type B1 is 15KB. In step S330, since the first data volume of log type B1 is greater than the first preset threshold, the log data belonging to log type B1 in the dataset X2 to be collected is removed, that is, log data A3 is removed from the dataset X2 to be collected.

[0094] Therefore, although the updated first data volume exceeds the first preset threshold during the Nth collection process, the restriction on log type B1 is only implemented during the N+1th collection process to minimize interference with the log collection process and maintain its stability.

[0095] Figure 5 A flowchart illustrating the sending of over-limit log information according to an embodiment of this disclosure is shown, such as... Figure 5 As shown, in some specific embodiments, when the first data volume of any log type meets the first preset condition, the step of removing log data corresponding to that log type from the dataset to be collected includes: step S331 and step S332.

[0096] In step S331, the first time of the log type in the suppressor is obtained, and the first time includes the time when the last over-limit log information was sent for the log type.

[0097] In step S332, it is determined that the interval between the first time and the current time meets a second preset condition. When the interval between the first time and the current time meets the second preset condition, an over-limit log message is sent according to the log type. When the interval between the first time and the current time does not meet the second preset condition, no over-limit log message is sent. Optionally, the second preset condition includes: the interval between the first time and the current time is greater than a third time interval.

[0098] For example, in the N+1th collection process, the dataset to be collected is X2, which includes log data A3 of log type B1. The size of the first data volume of log type B1 obtained from the rate limiter is 16KB, and the first preset threshold corresponding to log type B1 is 15KB. In step S330, since the first data volume of log type B1 is greater than the first preset threshold, log data A3 in the dataset to be collected X2 is removed. In step S331, the first time of log type B1 in the suppressor is obtained. Assuming that no over-limit log information has been sent for log type B1 before, the first time is empty. At this time, over-limit log information is sent to record the relevant content of limiting the collection of log data of log type B1.

[0099] In the (N+2)th data collection process, the dataset to be collected is X3, which includes log data A4. The log type of log data A4 is B1. The size of the first data of log type B1 obtained from the rate limiter is 16KB, and the first preset threshold corresponding to log type B1 is 15KB. Since the first data size of log type B1 is greater than the first preset threshold, log data A4 in the dataset to be collected X3 is removed, and the first time of log type B1 in the suppressor is obtained. Let the first preset time be M, where M is greater than the time interval between the (N+1)th and (N+2)th data collection processes. Since an over-limit log message has already been sent for log type B1 in the (N+1)th data collection process, no more over-limit log messages will be sent at this time, thereby reducing the frequency of sending over-limit log messages.

[0100] It should be noted that any descriptions not detailed in the embodiments of this disclosure can be found in the foregoing embodiments.

[0101] In some specific embodiments, the suppressor records multiple log types and the first time of each log type. The log collection method also includes step S410.

[0102] In step S410, for each log type in the suppressor, the first time of that log type is periodically reset according to the first time interval.

[0103] In the embodiments of this disclosure, the duration of the first time interval can be determined according to actual needs. For example, the duration of the first time interval can be in the millisecond, second, or even minute range.

[0104] Optionally, the duration of the first time interval is much longer than the refresh interval of the log directory to be collected. Specifically, the refresh interval of the log directory to be collected will be described in detail below, and will not be repeated here.

[0105] Optionally, the first time interval can be configured in the acquisition configuration file. After the current limiter is created, the first time interval is obtained by scanning the acquisition configuration file, and the suppressor is periodically reset according to the first time interval.

[0106] In some specific embodiments, the flow limiter records multiple log types and a first data volume for each log type, and the log collection method further includes step S510.

[0107] In step S510, for each log type in the rate limiter, the first data volume second time interval for that log type is periodically reset according to the second time interval. The first time interval and the second time interval have the same duration.

[0108] Optionally, for a log type, the rate limiter and the suppressor are reset synchronously. That is, when the first data volume of a log type in the rate limiter is reset, the first time of the log type in the suppressor is reset synchronously.

[0109] For example, as mentioned above, during the N+2th collection process, the first data volume of log type B1 exceeds the limit, that is, the first data volume is greater than the first preset threshold. Since the excess log information has already been sent for log type B1 during the N+1th collection process, the excess log information will not be sent again at this time.

[0110] In the N+m-th data collection process, the dataset to be collected is Xm, which includes log data Am. The log type of log data Am is B1, and the first preset threshold corresponding to log type B1 is 15KB. The first time of log type B1 in the suppressor is reset, and at the same time, the first data volume of log type B1 in the rate limiter is reset. At this time, the size of the first data volume of log type B1 obtained from the rate limiter is 0KB. Since the first data volume of log type B1 is less than the first preset threshold, log data Am can be retained. In other words, in the N+m-th data collection process, by resetting the suppressor and rate limiter, the previous restrictions on the collection of log data of log type B1 and the sending of logs exceeding the limit are lifted until the first data volume of log type B1 exceeds the first preset threshold again, at which point the collection of log data of log type B1 is restricted again.

[0111] In some specific embodiments, step S310 includes step S311.

[0112] In step S311, when the log directory to be collected is refreshed, the log data to be collected is obtained according to the directory to form the dataset to be collected. The durations of both the first time interval and the second time interval are greater than the refresh interval of the log directory to be collected. In embodiments of this disclosure, the refresh interval of the log directory to be collected refers to the interval between two consecutive refresh times of the log directory to be collected.

[0113] For example, when the log directory to be collected is refreshed for the Nth time, the Nth collection process described above is executed. After collecting log data A1, log type B1 and the first data volume of log type B1 are recorded in the rate limiter, and log type B1 is also recorded in the compression device. When the log directory to be collected is refreshed for the N+1th time, the N+1th collection process described above is executed. During this process, it is determined that the first data volume of log type B1 exceeds the limit, that is, the first data volume is greater than the first preset threshold. Therefore, an over-limit log message is sent for log type B1, and the time of sending the over-limit log message is recorded in the compression device as the first time. When the log directory to be collected is refreshed for the N+2th time, the N+2th collection process described above is executed. Since an over-limit log message has already been sent for log type B1 during the N+1th collection process, no more over-limit log messages are sent during the N+2th collection process. When the log directory to be collected is refreshed for the N+mth time, the N+mth collection process described above is executed, resetting the first time of log type B1 in the compressor, and simultaneously resetting the first data volume of log type B1 in the rate limiter. In other words, the duration of the first time interval is the same as the duration of the second time interval, both of which are approximately equal to the duration from the Nth refresh to the N+mth refresh of the log directory to be collected.

[0114] In some specific embodiments, the log collection method further includes step S610, step S620, and / or step S630.

[0115] In step S610, invalid log data in the target dataset is removed. The invalid log data includes log data that has not been collected after a preset time period.

[0116] In step S620, log types corresponding to invalid log data are removed from the flow limiter.

[0117] In step S630, log types corresponding to invalid log data are removed from the suppressor.

[0118] The duration of the first time interval and the duration of the first time interval are both less than the preset duration.

[0119] In this embodiment of the disclosure, the preset duration can be a relatively long duration, for example, the preset duration can be a day. That is, when a certain log data in the target dataset has not been collected for more than one day, the log data is confirmed to be invalid data.

[0120] For example, on day J, the Nth to N+mth data collection processes described above are executed. The difference is that, in the Nth data collection process, the dataset X1 to be collected includes log data A5 of log type B3. Since the amount of log data of type B3 is less than a first preset threshold, log data A5 in the dataset X1 to be collected is retained to obtain the target dataset X1'. Apart from this, the remaining content of the Nth to N+mth data collection processes can be found in the aforementioned embodiments and will not be repeated here.

[0121] On day J+1, it is determined that log data A5 in the target dataset X1' has not been collected for more than one day. Since a log type usually includes only one log data, log type B3 in the rate limiter and suppressor can be removed, thereby freeing up the storage space of the target dataset X1', the rate limiter, and the suppressor.

[0122] Based on the above log collection method, this disclosure also provides a log collection device for use at the collection source end. The following will combine... Figure 6 The device is described in detail.

[0123] Figure 6 A schematic diagram of a log collection apparatus according to an embodiment of the present disclosure is shown, such as... Figure 6 As shown, the log collection device 600 of this embodiment includes a first acquisition module 610, a second acquisition module 620, a processing module 630, and a collection module 640.

[0124] The first acquisition module 610 is used to acquire a dataset to be collected, which includes log data of at least one log type. In one embodiment, the first acquisition module 610 can be used to execute step S310 described above, which will not be repeated here.

[0125] The second acquisition module 620 is used to acquire a first data volume of each log type from the rate limiter for each log type. The first data volume includes the amount of log data of that log type that the rate limiter has currently collected. In one embodiment, the second acquisition module 620 can be used to execute step S320 described above, which will not be repeated here.

[0126] The processing module 630 is used to remove log data of that log type from the dataset to be collected when the first data volume of any log type meets the first preset condition, so as to obtain the target dataset. In one embodiment, the processing module 630 can be used to execute step S330 described above, which will not be repeated here.

[0127] The acquisition module 640 is used to acquire log data from the target dataset and send it to the cache channel. In one embodiment, the acquisition module 640 can be used to perform step S340 described above, which will not be repeated here.

[0128] The log collection device using the embodiments of this disclosure can perform rate limiting based on log type, thereby ensuring that the collection of log data of different log types is independent and does not affect each other. Furthermore, since the log collection method of the embodiments of this disclosure can filter data before encapsulating it into transactions, it also improves the problem of wasted resources at the data collection source.

[0129] According to embodiments of this disclosure, any plurality of modules among the first acquisition module 610, the second acquisition module 620, the processing module 630, and the acquisition module 640 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first acquisition module 610, the second acquisition module 620, the processing module 630, and the acquisition module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 610, the second acquisition module 620, the processing module 630, and the acquisition module 640 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0130] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a log collection method according to embodiments of the present disclosure, such as... Figure 7As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0131] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0132] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0133] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the log collection method according to the embodiments of this disclosure.

[0134] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0135] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the log collection method provided in the embodiments of this disclosure.

[0136] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0137] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0138] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0139] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0142] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A log collection method, characterized in that, Applied to the data acquisition source, including: When the log directory to be collected is refreshed, the log data to be collected is obtained according to the directory to be collected to obtain the dataset to be collected, which includes log data of at least one log type; For each log type, a first data volume for that log type is obtained from the rate limiter, the first data volume including: the data volume of log data of that log type that has been collected currently as recorded by the rate limiter; The rate limiter includes multiple rate limiting components, each of which records at least one log type of the first data volume, and different rate limiting components record different log types; When the first data volume of any log type meets the first preset condition, log data of that log type is removed from the dataset to be collected to obtain the target dataset. The removal of log data of that log type from the dataset to be collected includes: Get the first time of this log type in the suppressor, the first time including the time when the last over-limit log information was sent for this log type; When the interval between the first time and the current time meets the second preset condition, an over-limit log message is sent according to the log type. The first preset condition includes that the first data volume is greater than or equal to a first preset threshold, each log type corresponds to at least one first preset threshold, and the first preset thresholds corresponding to different log types may be the same or different; the second preset condition includes that the interval between the first time and the current time is greater than a preset third time interval. Collect log data from the target dataset and send it to the cache channel; The first data volume of at least one log type in the flow limiter is updated based on the size of each collected log data and the log type of each log data. Specifically, for each log type in the rate limiter, the first data size of that log type is periodically reset to 0 according to a second time interval, wherein the duration of the second time interval is greater than the refresh interval of the log directory to be collected, and the refresh interval of the log directory to be collected refers to the interval between two adjacent refresh times of the log directory to be collected; the suppressor records multiple log types and the first time of each log type; for each log type in the suppressor, the first time of that log type is periodically reset according to a first time interval, wherein the duration of the first time interval is the same as the duration of the second time interval; Specifically, for a log type, when the first data volume of that log type in the rate limiter is reset, the first time of that log type in the suppressor is also reset simultaneously.

2. The log collection method according to claim 1, characterized in that, The log collection method also includes: Remove invalid log data from the target dataset. The invalid log data includes log data that has not been collected after a preset time period. Remove the log type corresponding to the invalid log data from the rate limiter; and / or, remove the log type corresponding to the invalid log data from the suppressor; Wherein, the duration of both the first time interval and the second time interval is less than the preset duration.

3. A log collection device, characterized in that, Applied to the data acquisition source, including: The first acquisition module is used to acquire log data to be collected based on the log directory to be collected when the log directory to be collected is refreshed, and to obtain the dataset to be collected, wherein the dataset to be collected includes log data of at least one log type. The second acquisition module is used to acquire, for each log type, a first data volume of that log type from the rate limiter, wherein the first data volume includes: the data volume of log data of that log type that has been collected currently as recorded by the rate limiter; The rate limiter includes multiple rate limiting components, each of which records at least one log type of the first data volume, and different rate limiting components record different log types; The processing module is configured to remove log data of that log type from the dataset to be collected when the first data volume of any log type meets a first preset condition, in order to obtain a target dataset. The removal of log data of that log type from the dataset to be collected includes: Get the first time of this log type in the suppressor, the first time including the time when the last over-limit log information was sent for this log type; When the interval between the first time and the current time meets the second preset condition, an over-limit log message is sent according to the log type. The first preset condition includes that the first data volume is greater than or equal to a first preset threshold, each log type corresponds to at least one first preset threshold, and the first preset thresholds corresponding to different log types may be the same or different; the second preset condition includes that the interval between the first time and the current time is greater than a preset third time interval. The acquisition module is used to acquire log data from the target dataset and send it to the cache channel, and update the first data volume of at least one log type in the rate limiter according to the size of each acquired log data and the log type of each log data. Specifically, for each log type in the rate limiter, the first data size of that log type is periodically reset to 0 according to a second time interval, wherein the duration of the second time interval is greater than the refresh interval of the log directory to be collected, and the refresh interval of the log directory to be collected refers to the interval between two adjacent refresh times of the log directory to be collected; the suppressor records multiple log types and the first time of each log type; for each log type in the suppressor, the first time of that log type is periodically reset according to a first time interval, wherein the duration of the first time interval is the same as the duration of the second time interval; Specifically, for a log type, when the first data volume of that log type in the rate limiter is reset, the first time of that log type in the suppressor is also reset simultaneously.

4. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the log collection method according to claim 1 or 2.

5. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to perform the log collection method according to claim 1 or 2.

6. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the log collection method according to claim 1 or 2.

Images