Elastic scheduling method for log data acquisition

Abstract

The invention belongs to the technical field of network security, and particularly relates to an elastic scheduling method for log data acquisition, which comprises the following steps: acquiring configuration information corresponding to an acquisition node; initializing each collection node according to the configuration information so as to collect and analyze log data; in the log data acquisition process, monitoring data traffic processed by each acquisition node; and adaptively adjusting the size of a thread pool of the acquisition nodes according to the data traffic processed by each acquisition node, and / or adaptively adjusting the number of the acquisition nodes, so as to optimize the resource configuration of each acquisition node. The method is especially suitable for business requirements of safety product log collection, and can effectively cope with a large amount of fluctuating safety log data. Through dynamic resource scheduling, the reliability and efficiency of data acquisition are improved, and the robustness and expandability of the system are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a flexible scheduling method for log data collection. Background Art

[0002] With the increasing number of network security threats, log collection of security products has become an important means for enterprises and organizations to conduct security monitoring, incident response and compliance audits. When faced with large-scale log data, how to collect data efficiently and stably, while being able to flexibly respond to fluctuations in data traffic, has become a technical challenge.

[0003] Traditional data collection solutions often lack flexibility and cannot respond to sudden data growth in real time, which can easily cause data loss or system overload. Therefore, for the business scenario of security product log collection, there is an urgent need for a collection unit scheduling method that can dynamically adapt to data flow changes. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to propose a flexible scheduling method based on a microservice framework collection unit, aiming to solve the challenges brought about by data traffic fluctuations during security product log collection, and to dynamically adjust the system's resource allocation according to real-time data traffic to improve the efficiency and stability of data collection.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a flexible scheduling method for log data collection, characterized in that it includes: obtaining configuration information corresponding to the collection node; initializing each collection node according to the configuration information to collect and parse the log data; during the log data collection process, monitoring the data flow processed by each collection node; adaptively adjusting the thread pool size of the collection node according to the data flow processed by each collection node, and / or adaptively adjusting the number of collection nodes to optimize the resource configuration of each collection node.

[0006] According to a specific embodiment of the present invention, the steps of initializing each collection node according to the configuration information to collect and parse log data include: for each collection node, loading the corresponding local configuration information and remote configuration information to respectively extract the configuration information of the thread pool from the local configuration information and extract the configuration information of the data parsing rule from the remote configuration information; initializing the corresponding collection node according to the configuration information of the thread pool and the configuration information of the parsing rule to collect and parse log data.

[0007] According to a specific embodiment of the present invention, in the log data collection process, it includes: for each thread of the collection node, the collected log data is filtered and then written into the collection queue, and the total number of log data processed by the collection node is updated; wherein, when the total number of log data processed by the collection node exceeds a preset threshold, an alarm response is issued.

[0008] According to a specific embodiment of the present invention, the log data parsing process includes: for each thread of the collection node, reading and parsing the log data from the collection queue: if the log data parsing is successful, writing the log data to the storage; if the log data parsing fails, reporting an error and writing it to the cache queue.

[0009] According to a specific embodiment of the present invention, it also includes: regularly acquiring and loading remote configuration information to update the parsing rules configured by the collection node; wherein the collection node is also used to read and parse log data from the cache queue after the update.

[0010] According to a specific embodiment of the present invention, during the log data collection process, the step of monitoring the data flow processed by each collection node includes: for a collection node, calculating the change in the total amount of log data processed within a preset time period, and calculating the data flow processed by the collection node per unit time based on the change, so as to adaptively adjust the thread pool size of the collection node according to the data flow processed by the collection node per unit time, and / or adaptively adjust the number of collection nodes.

[0011] According to a specific embodiment of the present invention, the data flow processed by the collection node per unit time is calculated as follows: data flow processed per unit time=change in the total number of log data / preset time period.

[0012] According to a specific embodiment of the present invention, the step of adaptively adjusting the thread pool size of the collection node according to the data traffic processed by each collection node, and / or adaptively adjusting the number of collection nodes to optimize the resource configuration of each collection node includes: when the data traffic processed by the collection node per unit time is higher than a preset first threshold and lasts for more than a preset first time period, increasing the thread pool size of the collection node and / or adding a collection node with the same configuration as the collection node; when the data traffic processed by the collection node per unit time is lower than a preset second threshold and lasts for more than a preset second time period, reducing the thread pool size of the collection node and / or taking offline the collection node with the same configuration as the collection node.

[0013] According to a specific embodiment of the present invention, when the data flow processed by the collection node per unit time is higher than a preset first threshold value and the duration exceeds a preset first time period, the step of increasing the thread pool size of the collection node and / or adding a collection node with the same configuration as the collection node includes: when the data flow processed by the collection node per unit time is higher than a preset first threshold value and the duration exceeds the preset first time period, identifying the current thread pool size of the collection node; judging whether the current thread pool size exceeds the preset maximum value of the thread pool size: if so, additionally adding a collection node with the same configuration as the collection node to share the data flow processed by the collection node; if not, increasing the thread pool size of the collection node to devote more system resources to the collection node and expand the data flow it can process.

[0014] According to a specific embodiment of the present invention, when the data flow processed by the collection node per unit time is lower than a preset second threshold value and the duration exceeds a preset second time period, the step of reducing the thread pool size of the collection node and / or offline the collection node with the same configuration as the collection node includes: when the data flow processed by the collection node per unit time is lower than the preset second threshold value and the duration exceeds the preset second time period, identifying the current thread pool size of the collection node; judging whether the current thread pool size exceeds the preset middle value of the thread pool size: if so, reducing the thread pool size of the collection node to reclaim the system resources of the collection node; if not, offline the collection node with the same configuration as the collection node, or redeploying the collection node for different log data collection tasks.

[0015] The elastic scheduling method for log data collection provided by the present invention sets an appropriate number of core threads as the basic configuration during initialization, and simultaneously monitors traffic through timed tasks and dynamically adjusts the size of the thread pool to cope with sudden high concurrency situations, thereby ensuring the stability and performance of the system. The method is particularly suitable for the business needs of security product log collection, and can effectively cope with large amounts of fluctuating security log data. Through dynamic resource scheduling, not only the reliability and efficiency of data collection are improved, but also the robustness and scalability of the system are enhanced. In addition, an exception handling mechanism is added, which greatly improves the stability and security of the system, and is of great significance for network security management. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of a specific embodiment of a flexible scheduling method for log data collection provided by the present invention; Figure 2 A flow chart of another specific embodiment of a flexible scheduling method for log data collection provided by the present invention; Figure 3A structural schematic diagram of a specific embodiment of a flexible scheduling system for log data collection provided by the present invention; Figure 4 The present invention provides a structural block diagram of a specific embodiment of an electronic device. DETAILED DESCRIPTION

[0017] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0019] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0021] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0022] Example 1 The present invention provides a flexible scheduling method for log data collection, which is applied to a network data security monitoring system, and monitors the operation and maintenance status of a product object by acquiring its log data, so as to maintain the network security of the product object. Usually, a plurality of collection nodes are arranged in a network data security monitoring system to collect and parse the log data of a product object, and store it in a database, so that the operation status, data security, abnormal working conditions, etc. can be identified by analyzing the log data corresponding to the product object.

[0023] See also Figure 1 , as shown in 2, the elastic scheduling method for log data collection includes the following steps: Step S100, obtaining configuration information corresponding to the collection node.

[0024] First, before starting to execute the log data collection task, it is necessary to pre-configure each collection node so that it can perform the corresponding operation task, that is, complete the initialization of the collection node. To this end, the collection node needs to obtain and load the corresponding configuration information for initialization, and the configuration information includes local configuration information and remote configuration information, which correspond to the configuration information pre-stored in the collection node and the configuration information that needs to be called from the outside. It can be understood here that the local configuration information is already pre-stored in the collection node locally, so that it can be directly called and loaded from the local during initialization. In this embodiment, the local configuration information of the collection node may include, for example: thread pool configuration information, data filtering configuration information, and other configuration information related to data collection, and there is no excessive restriction on this.

[0025] It should also be noted that the thread pool is a form of multithreaded processing. During the processing, tasks are added to the queue and then automatically started after the thread is created. The thread pool maintains multiple threads, waiting for the supervisor to assign concurrently executable tasks, but the number of threads will never exceed the maximum. Threads exceeding the maximum can be queued, but they will not start until other threads are completed. The thread pool can not only ensure the full utilization of resources, but also prevent over-scheduling.

[0026] Therefore, the thread pool size of a collection node determines the number of log data collection tasks that can be executed simultaneously. For example, when the data flow is large, by increasing the thread pool size of the collection node, it can be able to collect log data of multiple different product objects at the same time to avoid resource loss; when the data flow is small, the thread pool size of the collection node can also be reduced to release system resources.

[0027] Remote configuration information is configuration information that the collection node needs to call from the outside. For example, since the network data security monitoring system needs to be responsible for managing the data security of different product objects, and the data format of each product object is different, after the product object is added to the system, it is necessary to uniformly query the corresponding data format to generate the corresponding data parsing rules. The collection node needs to obtain and load the configuration information about the parsing rules in order to parse the log data after collection, and finally store it in the database. Of course, remote configuration information is not limited to just parsing rules. For example, it can also include the number of parsing retries and other configuration information related to data parsing. There are also no excessive restrictions on this.

[0028] Step S200: Initialize each collection node according to the configuration information to collect and analyze log data. Based on the configuration information obtained above, the corresponding collection node can be configured according to it for initialization, that is, the thread pool size and data parsing rules of the collection node are configured accordingly, so that it can perform a certain amount of log data collection work at the same time, for example, it can collect log data of multiple product objects at the same time to cope with the scenario of data traffic peak, and can correctly parse the log data after the data collection is completed to store it in the database. It should be noted here that multiple collection nodes can load the same configuration information to achieve the same configuration, so as to form a cluster through containerized deployment, and then deploy multiple clusters to collect and parse different log data respectively, so as to achieve data security monitoring of all product objects. For example, a cluster only needs to be responsible for the collection and parsing of log data of one or more product objects, or the collection and parsing of a type of log data, and the collection and parsing of all log data is achieved through the division of labor and cooperation between different clusters. As for the containerized deployment of the collection node, it can be understood that the single-node deployment cannot cope with the peak of data traffic very well. Even if the thread pool size of the collection node is configured to the maximum, the rate at which it processes data cannot cope with the massive log data, resulting in data loss of the product object.

[0029] It should also be added that the collection node is not limited to only collecting and parsing the log data of multiple different product objects. It can be understood that based on the parsing rule configuration of the collection node, it can parse all the log data that can be correctly read, which can be different log data of a product object or log data of different product objects. There is no limitation on this. The containerized deployment of the collection node can cope with the peak of data traffic, but it does not necessarily require containerized deployment. For example, when the data traffic is small, a single node deployment can also be made. There are also no excessive restrictions on this. The modifications and embellishments made to the embodiments of the present invention by those skilled in the art without departing from the spirit of the present invention still fall within the scope of the invention application of the present invention.

[0030] Step S300, during the log data collection process, monitoring the data flow processed by each collection node.

[0031] Step S400 , adaptively adjusting the thread pool size of each collection node and / or adaptively adjusting the number of collection nodes according to the data flow processed by each collection node, so as to optimize the resource configuration of each collection node.

[0032] After the collection node is initialized, it can start collecting and parsing log data, that is, collecting log data of product objects and parsing them. Data collection and parsing are two different operations. First, the collection node needs to collect data first, and the counter needs to be initialized (used to calculate the total number of log data processed by the collection node), that is, all=0, and the total number of collected data at the last moment: lastAll=0. Secondly, the collection node can start collecting log data based on the configured thread pool size, where one thread corresponds to a log data collection task, and after the data is collected, the total number of processed log data is updated, that is, the total number of log data is recorded plus one (all=all+1), and the log data is written into the collection queue. Of course, after the data is collected and before it is written into the collection queue, the log data can also be preprocessed accordingly, such as data filtering, etc., without making too many restrictions on this. The modifications and embellishments made to the embodiments of the present invention by those skilled in the art without departing from the spirit of the present invention still fall within the scope of the invention patent application of the present invention.

[0033] Among them, during the data collection process, it is necessary to monitor the data flow processed by each collection node to identify the dynamic situation of the current data flow. For example, a thread of the collection node can be occupied to set a timing detection function, that is, to detect the data flow processed by the collection node once within a preset time period. Specifically, it is first necessary to identify the change in the total number of log data processed by the collection node between the previous moment and the current moment, current=all-lastAll, and further calculate the data flow processed by the collection node per unit time based on it, that is, TPS (flow processed per second) = change in the total number of log data / preset time period. Secondly, the TPS of the collection node is verified. If the TPS is higher than the preset first threshold and the duration exceeds the preset first time period, it means that the data flow currently carried by the collection node is large. In order to relieve the pressure of the node and avoid data loss, the thread pool size of the collection node can be increased, and / or a new collection node can be added to the cluster where the node is located to ensure the performance and stability of the collection.

[0034] It can be understood here that it is preferred to increase the thread pool size of the collection node first, and expand its capacity by adjusting system resources to cope with more data traffic. When the thread pool size of the collection node has reached the upper limit, it is meaningless to continue to increase the thread pool size. New collection nodes can be added to share the pressure of data traffic. Therefore, it can be determined first whether the thread pool size of the collection node has been adjusted to the preset maximum value. If so, an alarm can be issued to remind the operation and maintenance personnel to intervene, so as to add a new collection node. Otherwise, the thread pool size can be adjusted adaptively.

[0035] In a specific embodiment, the number of threads added to the thread pool can be calculated as follows: (current TPS-default TPS) / (default TPS / default thread pool size). It is understandable that the default thread pool size can be the number of threads when the acquisition node is initially configured, and the default TPS corresponds to the default number of threads. For example, the system can be stress tested in advance to obtain more reasonable initial configuration information so that the thread pool size initially configured at the acquisition node can cope with the data traffic under ideal conditions. However, in actual applications, the thread pool size can be adaptively adjusted based on the dynamic changes in the data traffic processed by the acquisition node to improve the flexibility and reliability of the network data security monitoring system.

[0036] Furthermore, when the number of threads that need to be added plus the original thread pool size exceeds the preset maximum value, new collection nodes can be added to cope with it.

[0037] Secondly, if the TPS is lower than the preset second threshold and lasts longer than the preset second time period, it means that the data traffic currently carried by the collection node is small and there is no great pressure. Accordingly, some system resources can be recovered, that is, the thread pool size of the collection node can be reduced, and / or some nodes in the cluster where the node is located can be taken offline (collection nodes with the same configuration as the collection node).

[0038] It can be understood that a middle value of the thread pool size can be pre-set as a metric. When the thread pool size exceeds the preset middle value, the thread pool size can be reduced first to avoid a single collection node occupying too many resources. If an abnormality occurs in the collection node, a large amount of data will be lost. When the thread pool size does not exceed the preset middle value, some of the collection nodes in the current cluster can be offline, or the collection nodes can be redeployed according to different log data collection tasks to reasonably allocate the resource usage in the system.

[0039] It can be seen that an appropriate number of core threads is set as the basic configuration during initialization, and the flow is monitored through scheduled tasks and the size of the thread pool is dynamically adjusted to cope with sudden high concurrency situations and ensure the stability and performance of the system. This ensures the effective use of system resources while providing sufficient processing power to cope with emergencies.

[0040] In addition, an exception handling mechanism is added during the execution of the above operations. For example, when the total amount of log data processed by any collection node exceeds the preset threshold, it means that the data traffic to be handled by the collection node is large, and an alarm response can be issued to remind the operation and maintenance personnel to make timely adjustments.

[0041] After the log data is collected, the collection node also needs to perform data analysis. One thread corresponds to a log data analysis task, that is, reading and parsing the log data from the collection queue. If the log data analysis is successful, the log data is written to the storage. If the log data analysis fails, an error is reported and the log data is written to the cache queue.

[0042] It should be noted here that parsing misplaced log data does not necessarily mean that the data is abnormal. It may be because a new product object has been added to the system, and the data format of the product object has not been configured in the collection node, which causes the collection node to be unable to correctly parse its log data. Therefore, if the data is directly considered abnormal and deleted, data loss will occur. In this regard, the collection node can first report an error for the log data that failed to parse and write it to the cache queue. At the same time, the collection node itself will regularly obtain and load new remote configuration information to update its configured data parsing rules. Then the collection node can pull the log data from the cache queue again and parse it after the parsing rules are updated, and finally store the successfully parsed data in the database. Of course, if the parsing still fails, then there is a high probability that the data is abnormal, and you can consider deleting it from the cache queue. The specific adjustment can be made according to the actual situation, and there are no too many restrictions on this.

[0043] It should also be added that the data collection and data analysis of the above-mentioned collection nodes can also be performed synchronously according to the preset thread pool size. Without making too many restrictions on this, the modifications and improvements made to the embodiments of the present invention by those skilled in the art without departing from the spirit of the present invention still fall within the scope of the invention patent application of the present invention.

[0044] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0045] Example 2 See also Figure 3 As shown, this embodiment also provides a flexible scheduling system for log data collection, including: The data loading module 10 is used to obtain the corresponding configuration information of the collection node.

[0046] The initialization module 20 initializes each collection node according to the configuration information to collect and analyze log data.

[0047] The flow monitoring module 30 monitors the data flow carried by each collection node during the log data collection process.

[0048] The dynamic adjustment module 40 adaptively adjusts the thread pool size of each collection node and / or the number of collection nodes according to the data traffic carried by each collection node, so as to optimize the resource configuration of each collection node.

[0049] It should be noted that the elastic scheduling system for log data collection provided in the above embodiment and the elastic scheduling method for log data collection provided in the above embodiment 1 belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In actual application, the elastic scheduling method for log data collection provided in the above embodiment 1 can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0050] Example 3 See also Figure 4 As shown, an embodiment of the present application further provides an electronic device, comprising a memory 2, a processor 1, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above methods are implemented.

[0051] Among them, the memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory can also include both an internal storage unit of the electronic device and an external storage device. The memory can not only be used to store application software and various types of data installed in the electronic device, but also can be used to temporarily store data that has been output or is to be output.

[0052] In some embodiments, the processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor is the control core (Control Unit) of the electronic device, which connects the various components of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory, and calling data stored in the memory.

[0053] The processor executes the operating system of the electronic device and various installed application programs. The processor executes the application programs to implement the steps in the above method embodiment.

[0054] Exemplarily, the program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of program instruction segments capable of completing specific functions, which are used to describe the execution process of the program in the electronic device.

[0055] The above-mentioned integrated unit implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium and include several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of various embodiments of the present invention.

[0056] In summary, the elastic scheduling method for log data collection provided by the present invention sets an appropriate number of core threads as the basic configuration during initialization, and at the same time monitors traffic through scheduled tasks and dynamically adjusts the size of the thread pool to cope with sudden high concurrency situations, ensuring the stability and performance of the system, and is particularly suitable for the business needs of security product log collection, and can effectively cope with large amounts of fluctuating security log data. Through dynamic resource scheduling, not only the reliability and efficiency of data collection are improved, but also the robustness and scalability of the system are enhanced. In addition, an exception handling mechanism is added, which greatly improves the stability and security of the system, which is of great significance for network security management.

[0057] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A flexible scheduling method for log data collection, characterized in that: include: Get the corresponding configuration information of the collection node; Initialize each collection node according to the configuration information to collect and analyze log data; During the log data collection process, monitor the data flow processed by each collection node; The thread pool size of each collection node is adaptively adjusted according to the data flow processed by each collection node, and / or the number of collection nodes is adaptively adjusted to optimize the resource configuration of each collection node.

2. The elastic scheduling method for log data collection according to claim 1, characterized in that: The steps of initializing each collection node according to the configuration information to collect and analyze log data include: For each acquisition node, load the corresponding local configuration information and remote configuration information, so as to respectively extract the configuration information of the thread pool from the local configuration information and extract the configuration information of the data parsing rule from the remote configuration information; The corresponding collection node is initialized according to the configuration information of the thread pool and the configuration information of the parsing rule to collect and parse the log data.

3. The elastic scheduling method for log data collection according to claim 1 is characterized in that: The log data collection process includes: For each thread of the collection node, the collected log data is filtered and written into the collection queue, and the total number of log data processed by the collection node is updated; Wherein, when the total amount of log data processed by the collection node exceeds a preset threshold, an alarm response is issued.

4. The elastic scheduling method for log data collection according to claim 2, characterized in that: The log data parsing process includes: For each thread of the collection node, read and parse the log data from the collection queue: If the log data is parsed successfully, the log data is written to the storage; If the log data parsing fails, an error is reported and the log data is written to the cache queue.

5. The elastic scheduling method for log data collection according to claim 4 is characterized in that: Also includes: Regularly obtain and load remote configuration information to update the parsing rules of the collection node configuration; After being updated, the collection node is also used to read and parse log data from the cache queue.

6. The elastic scheduling method for log data collection according to claim 1, characterized in that: During the log data collection process, the steps to monitor the data traffic processed by each collection node include: For a collection node, the change in the total amount of log data processed within a preset time period is calculated, and the data flow processed by the collection node per unit time is calculated based on it, so as to adaptively adjust the thread pool size of the collection node and / or adaptively adjust the number of collection nodes according to the data flow processed by the collection node per unit time.

7. The elastic scheduling method for log data collection according to claim 6, characterized in that: The data flow processed by the collection node per unit time is calculated as follows: Data flow processed per unit time = change in total log data / preset time period.

8. The elastic scheduling method for log data collection according to claim 1, characterized in that: The steps of adaptively adjusting the thread pool size of each collection node and / or adaptively adjusting the number of collection nodes according to the data flow processed by each collection node to optimize the resource configuration of each collection node include: When the data flow processed by the acquisition node per unit time is higher than a preset first threshold value and the duration exceeds a preset first time period, increase the thread pool size of the acquisition node and / or add an acquisition node with the same configuration as the acquisition node; When the data flow processed by the collection node per unit time is lower than a preset second threshold and lasts longer than a preset second time period, the thread pool size of the collection node is reduced, and / or a collection node with the same configuration as the collection node is offline.

9. The elastic scheduling method for log data collection according to claim 8, characterized in that: When the data flow processed by the acquisition node per unit time is higher than a preset first threshold value and the duration exceeds a preset first time period, the step of increasing the thread pool size of the acquisition node and / or adding an acquisition node with the same configuration as the acquisition node includes: When the data flow processed by the collection node per unit time is higher than a preset first threshold and the duration exceeds a preset first time period, identifying the current thread pool size of the collection node; Determine whether the current thread pool size exceeds the preset maximum value of the thread pool size: If yes, then add an additional collection node with the same configuration as the collection node to share the data traffic processed by the collection node If not, increase the thread pool size of the acquisition node to devote more system resources to the acquisition node and expand the data flow that it can process.

10. The elastic scheduling method for log data collection according to claim 8, characterized in that: When the data flow processed by the acquisition node per unit time is lower than a preset second threshold value and the duration exceeds a preset second time period, the steps of reducing the thread pool size of the acquisition node and / or offline the acquisition node with the same configuration as the acquisition node include: When the data flow processed by the collection node per unit time is lower than a preset second threshold value and the duration exceeds a preset second time period, identifying the current thread pool size of the collection node; Determine whether the current thread pool size exceeds the preset middle value of the thread pool size: If yes, reducing the thread pool size of the acquisition node to recycle system resources of the acquisition node; If not, take offline a collection node with the same configuration as the collection node, or redeploy a collection node for a different log data collection task.

Images