A flexible scheduling method for log data collection

Abstract

The present invention belongs to the field of network security technology, and specifically relates to a flexible scheduling method for log data collection, comprising: obtaining configuration information corresponding to collection nodes; initializing each collection node according to the configuration information to collect and parse log data; monitoring the data flow processed by each collection node during the log data collection process; and adaptively adjusting the thread pool size of the collection node and / or the number of collection nodes according to the data flow processed by each collection node to optimize the resource allocation of each collection node. The present invention is particularly suitable for the business needs of security product log collection and can effectively cope with large amounts of fluctuating security log data. Through dynamic resource scheduling, not only the reliability and efficiency of data collection are improved, but also the robustness and scalability of the system are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a flexible scheduling method for log data collection. Background Art

[0002] With the increasing threat of cybersecurity threats, security product log collection has become a crucial tool for businesses and organizations to conduct security monitoring, incident response, and compliance audits. Faced with large amounts of log data, efficient and stable data collection, while also flexibly responding to fluctuations in data traffic, presents a technical challenge.

[0003] Traditional data collection solutions often lack flexibility and are unable to immediately respond to sudden data growth, which can easily lead to data loss or system overload. Therefore, for the business scenario of security product log collection, there is an urgent need for a collection unit scheduling method that can dynamically adapt to changes in data traffic. Summary of the Invention

[0004] In view of the shortcomings of the existing technology mentioned above, the purpose of the present invention is to propose a flexible scheduling method based on a microservice framework collection unit, aiming to solve the challenges brought about by data traffic fluctuations during the security product log collection process, and to dynamically adjust the system's resource allocation according to real-time data traffic, thereby improving the efficiency and stability of data collection.

[0005] To achieve the above-mentioned and other related purposes, the present invention provides a flexible scheduling method for log data collection, which is characterized by including: obtaining configuration information corresponding to the collection node; initializing each collection node according to the configuration information to collect and parse log data; monitoring the data traffic processed by each collection node during the log data collection process; adaptively adjusting the thread pool size of the collection node and / or adaptively adjusting the number of collection nodes according to the data traffic processed by each collection node to optimize the resource configuration of each collection node.

[0006] According to a specific embodiment of the present invention, the steps of initializing each collection node according to the configuration information to collect and parse log data include: for each collection node, loading the corresponding local configuration information and remote configuration information to respectively extract the configuration information of the thread pool from the local configuration information and the configuration information of the data parsing rule from the remote configuration information; initializing the corresponding collection node according to the configuration information of the thread pool and the configuration information of the parsing rule to collect and parse log data.

[0007] According to a specific embodiment of the present invention, during the log data collection process, it includes: for each thread of the collection node, the collected log data is filtered and written into the collection queue, and the total number of log data processed by the collection node is updated; wherein, when the total number of log data processed by the collection node exceeds a preset threshold, an alarm response is issued.

[0008] According to a specific embodiment of the present invention, the log data parsing process includes: for each thread of the collection node, reading and parsing the log data from the collection queue: if the log data parsing is successful, the log data is written to the storage; if the log data parsing fails, an error is reported and the log data is written to the cache queue.

[0009] According to a specific embodiment of the present invention, the method further includes: periodically acquiring and loading remote configuration information to update parsing rules configured in the collection node; wherein, after the update, the collection node is further used to read and parse log data from the cache queue.

[0010] According to a specific embodiment of the present invention, during the log data collection process, the step of monitoring the data flow processed by each collection node includes: for a collection node, calculating the change in the total amount of log data processed within a preset time period, and calculating the data flow processed by the collection node per unit time based on the change, so as to adaptively adjust the thread pool size of the collection node and / or adaptively adjust the number of collection nodes according to the data flow processed by the collection node per unit time.

[0011] According to a specific embodiment of the present invention, the data flow processed by the collection node per unit time is calculated as follows: data flow processed per unit time = change in the total number of log data / preset time period.

[0012] According to a specific embodiment of the present invention, the step of adaptively adjusting the thread pool size of the collection node and / or adaptively adjusting the number of collection nodes according to the data traffic processed by each collection node to optimize the resource configuration of each collection node includes: when the data traffic processed by the collection node per unit time is higher than a preset first threshold and the duration exceeds a preset first time period, increasing the thread pool size of the collection node and / or adding a collection node with the same configuration as the collection node; when the data traffic processed by the collection node per unit time is lower than a preset second threshold and the duration exceeds a preset second time period, reducing the thread pool size of the collection node and / or taking offline the collection node with the same configuration as the collection node.

[0013] According to a specific embodiment of the present invention, when the data flow processed by the collection node per unit time is higher than a preset first threshold value and the duration exceeds a preset first time period, the step of increasing the thread pool size of the collection node and / or adding a collection node with the same configuration as the collection node includes: when the data flow processed by the collection node per unit time is higher than the preset first threshold value and the duration exceeds the preset first time period, identifying the current thread pool size of the collection node; judging whether the current thread pool size exceeds the preset maximum value of the thread pool size: if so, adding an additional collection node with the same configuration as the collection node to share the data flow processed by the collection node; if not, increasing the thread pool size of the collection node to devote more system resources to the collection node and expand the data flow it can process.

[0014] According to a specific embodiment of the present invention, when the data flow processed by the collection node per unit time is lower than a preset second threshold value and the duration exceeds a preset second time period, the steps of reducing the thread pool size of the collection node and / or offline the collection node with the same configuration as the collection node include: when the data flow processed by the collection node per unit time is lower than the preset second threshold value and the duration exceeds the preset second time period, identifying the current thread pool size of the collection node; judging whether the current thread pool size exceeds the preset middle value of the thread pool size: if so, reducing the thread pool size of the collection node to reclaim the system resources of the collection node; if not, offline the collection node with the same configuration as the collection node, or redeploying the collection node for different log data collection tasks.

[0015] The flexible scheduling method for log data collection provided by the present invention sets an appropriate number of core threads as the basic configuration during initialization. It also monitors traffic through scheduled tasks and dynamically adjusts the thread pool size to cope with sudden high-concurrency situations, ensuring system stability and performance. This method is particularly suitable for the business needs of security product log collection and can effectively handle large amounts of fluctuating security log data. Dynamic resource scheduling not only improves the reliability and efficiency of data collection, but also enhances the robustness and scalability of the system. Furthermore, an exception handling mechanism is added, significantly improving the stability and security of the system, which is of great significance to network security management. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of a specific embodiment of a flexible scheduling method for log data collection provided by the present invention;

[0017] Figure 2 This is a flow chart of another specific embodiment of a flexible scheduling method for log data collection provided by the present invention;

[0018] Figure 3 This is a structural diagram of a specific embodiment of a flexible scheduling system for log data collection provided by the present invention;

[0019] Figure 4 This is a structural block diagram of a specific embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] To facilitate understanding of the present application, the present application will be described more fully below with reference to the accompanying drawings. The accompanying drawings provide embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of the present application more thorough and comprehensive.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0022] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0023] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0024] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0025] Example 1

[0026] The present invention provides a flexible scheduling method for log data collection, applicable to network data security monitoring systems. This method monitors the operational status of product objects by acquiring their log data, thereby maintaining the network security of these objects. Typically, network data security monitoring systems deploy multiple collection nodes to collect and parse the log data of product objects and store it in a database. This allows subsequent analysis of the log data corresponding to these product objects to identify their operational status, data security, operational anomalies, and so on.

[0027] See Figure 1 As shown in Figure 2, the elastic scheduling method for log data collection includes the following steps:

[0028] Step S100: Obtain configuration information corresponding to the collection node.

[0029] First, before starting to execute the log data collection task, it is necessary to pre-configure each collection node so that it can perform the corresponding operation task, that is, complete the initialization of the collection node. To this end, the collection node needs to obtain and load the corresponding configuration information for initialization, and the configuration information includes local configuration information and remote configuration information, which respectively correspond to the configuration information pre-stored in the collection node and the configuration information that needs to be called from the outside. It can be understood here that the local configuration information is already pre-stored locally in the collection node, so that it can be directly called and loaded from the local during initialization. In this embodiment, the local configuration information of the collection node can, for example, include: thread pool configuration information, data filtering configuration information, and other configuration information related to data collection, and there are no excessive restrictions on this.

[0030] It's also worth noting that a thread pool is a form of multithreaded processing that adds tasks to a queue and automatically starts them after a thread is created. The thread pool maintains multiple threads, waiting for the supervisor to assign concurrently executable tasks. However, the number of threads never exceeds the maximum. Threads exceeding the maximum can be queued, but they won't start until other threads complete. The thread pool not only ensures full resource utilization but also prevents overscheduling.

[0031] Therefore, the thread pool size of a collection node determines the number of log data collection tasks that can be executed simultaneously. For example, when the data flow is large, increasing the thread pool size of the collection node can enable it to collect log data of multiple different product objects at the same time to avoid resource loss; when the data flow is small, the thread pool size of the collection node can also be reduced to release system resources.

[0032] Remote configuration information is configuration information that the collection node needs to call from the outside. For example, since the network data security monitoring system is responsible for managing the data security of different product objects, and the data format of each product object is different, after adding the product object to the system, it is necessary to uniformly query the corresponding data format to generate the corresponding data parsing rules. The collection node needs to obtain and load the configuration information about the parsing rules in order to parse the log data after collection, and finally store it in the database. Of course, remote configuration information is not limited to just parsing rules. For example, it can also include the number of parsing retries and other configuration information related to data parsing. There are also no excessive restrictions on this.

[0033] Step S200: Initialize each collection node according to the configuration information to collect and analyze log data.

[0034] Based on the configuration information obtained above, the corresponding collection node can be configured accordingly for initialization. That is, the thread pool size and data parsing rules of the collection node are configured accordingly so that it can simultaneously perform a certain amount of log data collection work, for example, it can simultaneously collect log data from multiple product objects to cope with data traffic peaks, and after data collection, it can correctly parse the log data and store it in the database. It should be noted that multiple collection nodes can load the same configuration information to achieve the same configuration, thereby forming a cluster through containerized deployment. Then, by deploying multiple clusters, different log data collection and parsing can be performed separately to achieve data security monitoring for all product objects. For example, a cluster is only responsible for collecting and parsing log data for one or more product objects, or collecting and parsing one type of log data, and the collection and parsing of all log data can be achieved through division of labor and cooperation between different clusters. However, it is understandable that single-node deployment cannot effectively cope with data traffic peaks. Even if the thread pool size of the collection node is configured to the maximum, its data processing rate cannot cope with the massive amount of log data, resulting in data loss of product objects.

[0035] It should also be added that the collection node is not limited to just collecting and parsing the log data of multiple different product objects. It can be understood that based on the parsing rule configuration of the collection node, it can parse all log data that can be correctly read, which can be different log data of a product object or log data of different product objects. There is no limitation on this. The containerized deployment of the collection node can cope with the situation of peak data traffic, but it does not necessarily require containerized deployment. For example, when the data traffic is low, a single node deployment can also be used. There are also no excessive restrictions on this. The modifications and embellishments made by those skilled in the art to the embodiments of the present invention without departing from the spirit of the present invention still fall within the scope of the invention application of the present invention.

[0036] Step S300 , during the log data collection process, monitor the data flow processed by each collection node.

[0037] Step S400 : adaptively adjusting the thread pool size of each collection node and / or adaptively adjusting the number of collection nodes according to the data flow processed by each collection node, so as to optimize the resource configuration of each collection node.

[0038] After initialization, the collection node can begin collecting and parsing log data, specifically collecting and parsing log data from product objects. Data collection and parsing are two distinct operations. First, the collection node must first collect data, initializing a counter (used to calculate the total number of log data processed by the collection node) to all = 0, and the total number of collected data as of the previous moment: lastAll = 0. Second, the collection node can begin collecting log data based on the configured thread pool size, with each thread corresponding to a log data collection task. After data collection is complete, the total number of processed log data is updated by adding one to the total number of log data (all = all + 1), and the log data is written to the collection queue. Of course, after data collection is complete and before writing to the collection queue, the log data can also undergo appropriate data preprocessing, such as data filtering. Without limiting this, modifications and enhancements to the embodiments of the present invention made by those skilled in the art without departing from the spirit of the present invention remain within the scope of the present invention.

[0039] During the data collection process, it is necessary to monitor the data flow processed by each collection node to identify the current dynamics of data flow. For example, a thread on a collection node can be used to set up a timed detection function, which monitors the data flow processed by the collection node once within a preset time period. Specifically, the change in the total number of log data processed by the collection node between the previous and current time periods must be identified: current = all - lastAll. This change can then be used to calculate the data flow processed by the collection node per unit time: TPS (transactions per second) = change in total number of log data / preset time period. Next, the collection node's TPS is verified. If the TPS exceeds a preset first threshold and persists for longer than the preset first time period, it indicates that the collection node is currently carrying a high amount of data flow. To alleviate pressure on the node and prevent data loss, the collection node's thread pool size can be increased and / or new collection nodes can be added to the cluster in which the node resides to ensure collection performance and stability.

[0040] It's understandable that it's best to first increase the thread pool size of the collection node, expanding its capacity by adjusting system resources to handle increased data traffic. However, if the collection node's thread pool size has reached its upper limit, further increases are pointless, and new collection nodes can be added to share the data traffic pressure. Therefore, it's possible to first determine whether the collection node's thread pool size has been adjusted to the preset maximum. If so, an alarm can be triggered to alert operations and maintenance personnel to intervene and add new collection nodes. Otherwise, the thread pool size can be adaptively adjusted.

[0041] In one specific embodiment, the number of threads added to the thread pool can be calculated as follows: (current TPS - default TPS) / (default TPS / default thread pool size). It is understood that the default thread pool size can be the number of threads initially configured for the collection node, and the default TPS corresponds to the default number of threads. For example, a system can be stress-tested in advance to obtain reasonable initial configuration information, ensuring that the thread pool size initially configured for the collection node can handle ideal data traffic. However, in actual applications, the thread pool size of the collection node can be adaptively adjusted based on dynamic changes in the data traffic processed by the collection node, thereby improving the flexibility and reliability of the network data security monitoring system.

[0042] Furthermore, when the number of threads that need to be increased plus the original thread pool size exceeds the preset maximum value, new collection nodes can be added to cope with it.

[0043] Secondly, if the TPS is lower than the preset second threshold and lasts longer than the preset second time period, it means that the data traffic currently carried by the collection node is small and there is no great pressure. Accordingly, some system resources can be recovered, that is, the thread pool size of the collection node can be reduced, and / or some nodes in the cluster where the node is located can be taken offline (collection nodes with the same configuration as the collection node).

[0044] It's understandable that a pre-set intermediate thread pool size can be used as a benchmark. When the thread pool size exceeds this threshold, it can be prioritized to reduce it to avoid a single collection node occupying too many resources. If this collection node experiences an exception, a large amount of data will be lost. When the thread pool size does not exceed this threshold, some collection nodes in the current cluster can be taken offline, or redeployed according to different log data collection tasks to properly allocate system resources.

[0045] As can be seen from this, setting an appropriate number of core threads as the basic configuration during initialization, and using scheduled tasks to monitor traffic and dynamically adjust the thread pool size to cope with sudden high concurrency situations and ensure system stability and performance. This ensures efficient use of system resources while providing sufficient processing power to cope with emergencies.

[0046] In addition, an exception handling mechanism has been added during the execution of the above operations. For example, when the total amount of log data processed by any collection node exceeds the preset threshold, it means that the data traffic handled by the collection node is large, and an alarm response can be issued to remind the operation and maintenance personnel to make timely adjustments.

[0047] After the log data is collected, the collection node also needs to perform data analysis. One thread corresponds to a log data analysis task, that is, reading and parsing the log data from the collection queue. If the log data is parsed successfully, the log data is written to the storage. If the log data parsing fails, an error is reported and the log data is written to the cache queue.

[0048] It's important to note that parsing errors in log data don't necessarily indicate a data anomaly. It could be because a new product object has been added to the system, and the data format for that product object hasn't yet been configured in the collection node, preventing the collection node from correctly parsing the log data. Therefore, simply deleting the data as an anomaly can result in data loss. To address this, the collection node can first report an error for log data that failed parsing and write it to a cache queue. Simultaneously, the collection node itself periodically retrieves and loads new remote configuration information, updating its configured data parsing rules. After the parsing rules are updated, the collection node can then pull log data from the cache queue again, parse it, and finally store the successfully parsed data in the database. Of course, if parsing still fails, it's likely that the data is anomaly, and you can consider deleting it from the cache queue. This can be adjusted based on your specific situation, and there are no specific restrictions on this.

[0049] It should also be added that the data collection and data analysis of the above-mentioned collection nodes can also be performed synchronously according to the preset thread pool size. Without making too many restrictions on this, the modifications and improvements made to the embodiments of the present invention by those skilled in the art without departing from the spirit of the present invention still fall within the scope of the invention patent application of the present invention.

[0050] It should be noted that the step division of the various methods above is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0051] Example 2

[0052] See Figure 3 As shown, this embodiment also provides a flexible scheduling system for log data collection, including:

[0053] The data loading module 10 is used to obtain the corresponding configuration information of the collection node.

[0054] The initialization module 20 initializes each collection node according to the configuration information to collect and analyze log data.

[0055] The traffic monitoring module 30 monitors the data traffic carried by each collection node during the log data collection process.

[0056] The dynamic adjustment module 40 adaptively adjusts the thread pool size of each collection node and / or the number of collection nodes according to the data traffic carried by each collection node, so as to optimize the resource configuration of each collection node.

[0057] It should be noted that the elastic scheduling system for log data collection provided in the above embodiment and the elastic scheduling method for log data collection provided in Example 1 above are based on the same concept. The specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In actual applications, the elastic scheduling method for log data collection provided in Example 1 above can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0058] Example 3

[0059] See Figure 4 As shown, an embodiment of the present application further provides an electronic device, comprising a memory 2, a processor 1, and a program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0060] The memory includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory can also include both an internal storage unit of the electronic device and an external storage device. The memory can be used not only to store application software installed in the electronic device and various types of data, but also to temporarily store data that has been output or is about to be output.

[0061] In some embodiments, the processor may be comprised of an integrated circuit, such as a single packaged integrated circuit or multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor is the control core (control unit) of the electronic device, connecting the various components of the electronic device using various interfaces and circuits. It executes programs or modules stored in the memory and accesses data stored in the memory to perform various functions of the electronic device and process data.

[0062] The processor executes the operating system of the electronic device and various installed application programs. The processor executes the application programs to implement the steps in the above method embodiment.

[0063] Exemplarily, the program may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the program in the electronic device.

[0064] The above-mentioned integrated units implemented in the form of software functional modules can be stored in a computer-readable storage medium. The above-mentioned software functional modules stored in a storage medium include a number of instructions for causing a computer device (which may be a personal computer, computer equipment, or network equipment, etc.) or a processor to perform some of the functions of various embodiments of the present invention.

[0065] In summary, the flexible scheduling method for log data collection provided by the present invention sets an appropriate number of core threads as the basic configuration during initialization. At the same time, it monitors traffic through scheduled tasks and dynamically adjusts the size of the thread pool to cope with sudden high concurrency situations, ensuring the stability and performance of the system. It is particularly suitable for the business needs of security product log collection and can effectively handle large amounts of fluctuating security log data. Through dynamic resource scheduling, not only the reliability and efficiency of data collection are improved, but also the robustness and scalability of the system are enhanced. In addition, an exception handling mechanism is added, which greatly improves the stability and security of the system, which is of great significance for network security management.

[0066] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A flexible scheduling method for log data collection, characterized in that: include: Get the corresponding configuration information of the collection node; Initialize each collection node according to the configuration information to collect and analyze log data; During the log data collection process, monitor the data traffic processed by each collection node; Adaptively adjust the thread pool size of each collection node and the number of collection nodes based on the data traffic processed by each collection node to optimize the resource allocation of each collection node, including: When the data flow processed by the collection node per unit time is higher than a preset first threshold and lasts longer than a preset first time period, identify the current thread pool size of the collection node and determine whether the current thread pool size exceeds a preset maximum value of the thread pool size: If so, additional collection nodes with the same configuration as the collection node are added to share the data traffic processed by the collection node; If not, increase the thread pool size of the collection node to allocate more system resources to the collection node and expand the data flow it can process; wherein, when the thread pool of the collection node exceeds the preset maximum value after the increase, add additional collection nodes with the same configuration as the collection node; When the data flow processed by the collection node per unit time is lower than a preset second threshold and lasts for more than a preset second time period, the thread pool size of the collection node is reduced, and / or the collection node with the same configuration as the collection node is offline.

2. The elastic scheduling method for log data collection according to claim 1, characterized in that: The steps of initializing each collection node according to the configuration information to collect and analyze log data include: For each collection node, load the corresponding local configuration information and remote configuration information, so as to extract the configuration information of the thread pool from the local configuration information and the configuration information of the data parsing rule from the remote configuration information; Initialize the corresponding collection node according to the configuration information of the thread pool and the configuration information of the parsing rule to collect and parse the log data.

3. The elastic scheduling method for log data collection according to claim 1, characterized in that: The log data collection process includes: For each thread of the collection node, the collected log data is filtered and written to the collection queue, and the total number of log data processed by the collection node is updated; When the total amount of log data processed by the collection node exceeds a preset threshold, an alarm response is triggered.

4. The elastic scheduling method for log data collection according to claim 2, characterized in that: The log data parsing process includes: For each thread of the collection node, read and parse log data from the collection queue: If the log data is parsed successfully, the log data is written to the storage; If log data parsing fails, an error is reported and the data is written to the cache queue.

5. The elastic scheduling method for log data collection according to claim 4, characterized in that: Also includes: Regularly obtain and load remote configuration information to update the parsing rules of the collection node configuration; After being updated, the collection node is further used to read and parse log data from the cache queue.

6. The elastic scheduling method for log data collection according to claim 1, characterized in that: During the log data collection process, the steps for monitoring the data traffic processed by each collection node include: For a collection node, the change in the total amount of log data processed within a preset time period is calculated, and the data flow processed by the collection node per unit time is calculated based on it, so as to adaptively adjust the thread pool size of the collection node and / or adaptively adjust the number of collection nodes according to the data flow processed by the collection node per unit time.

7. The elastic scheduling method for log data collection according to claim 6, characterized in that: The data flow processed by the collection node per unit time is calculated as follows: The data volume processed per unit time = the change in the total amount of log data / the preset time period.

8. The elastic scheduling method for log data collection according to claim 1, characterized in that: When the data flow processed by the collection node per unit time is lower than a preset second threshold and lasts for more than a preset second time period, the steps of reducing the thread pool size of the collection node and / or offline a collection node with the same configuration as the collection node include: When the data flow processed by the collection node per unit time is lower than a preset second threshold and the duration exceeds a preset second time period, identifying the current thread pool size of the collection node; Determine whether the current thread pool size exceeds the preset middle value of the thread pool size: If so, reducing the thread pool size of the acquisition node to reclaim system resources of the acquisition node; If not, take offline a collection node with the same configuration as the collection node, or redeploy a collection node for a different log data collection task.

Images