Log collection system load adjustment method, device, equipment and storage medium
By acquiring stress metrics from the elastic search cluster, determining the load status, and adjusting the batch size parameters of the log collection and processing components, the system addresses the issues of insufficient stability and throughput under high pressure, achieving adaptive rate limiting and improved system stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2026-02-11
- Publication Date
- 2026-06-30
AI Technical Summary
The existing log collection system cannot dynamically adjust according to the real-time load status of the elastic search cluster, resulting in insufficient stability and throughput under high pressure.
By obtaining stress metrics information from the elastic search cluster, the load status is determined, and the batch size parameters of the log collection and processing components are adjusted according to the load status. The configuration file is then updated to achieve adaptive rate limiting, thereby avoiding overload and service interruption of the log collection system.
The system achieved adaptive adjustment of the log collection system, which improved the system's stability and throughput, and reduced manual maintenance costs.
Smart Images
Figure CN122309274A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data, and in particular to a load adjustment method, apparatus, device and storage medium for a log collection system. Background Technology
[0002] In large-scale distributed systems, log collection and storage are core components for ensuring system stability, troubleshooting, and performance optimization. With the deepening of enterprise IT infrastructure development, the number and scale of application systems that log centers need to connect to are growing exponentially, leading to a dramatic increase in log data volume.
[0003] Existing log collection systems typically use statically configured log collector parameters to control the rate at which data is written to the Elastic Search cluster. However, this approach cannot dynamically adjust based on the real-time load status of the Elastic Search cluster, resulting in a failure to guarantee the stability and throughput of the log collection system under high pressure.
[0004] Therefore, existing log collection systems suffer from insufficient stability. Summary of the Invention
[0005] This application provides a load adjustment method, apparatus, device, and storage medium for a log collection system, in order to solve the technical problem of insufficient stability in existing log collection systems.
[0006] Firstly, this application provides a load adjustment method for a log collection system, including:
[0007] Obtain stress metrics information for the elastic search cluster in the log collection system;
[0008] Determine the load status of the elastic search cluster based on stress indicator information;
[0009] Adjust the current batch size parameter of the log collection and processing component in the log collection system according to the load status to obtain the target batch size parameter;
[0010] Update the configuration file of the log collection and processing component based on the target batch size parameter, so that the log collection and processing component can perform log collection for the Elastic Search cluster according to the updated configuration file.
[0011] Secondly, this application provides a load adjustment device for a log collection system, comprising:
[0012] The acquisition module is used to acquire stress indicator information of the elastic search cluster in the log collection system.
[0013] The determination module is used to determine the load status of the elastic search cluster based on stress indicator information;
[0014] The adjustment module is used to adjust the current batch size parameter of the log collection and processing component in the log collection system according to the load status in order to obtain the target batch size parameter.
[0015] The update module is used to update the configuration file of the log collection and processing component according to the target batch size parameter, so that the log collection and processing component can perform log collection for the Elastic Search cluster according to the updated configuration file.
[0016] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0017] The memory stores instructions that the computer executes;
[0018] The processor executes computer execution instructions stored in memory to implement the first aspect or various possible implementations of the first aspect as described above.
[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect or various possible embodiments thereof.
[0020] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect or various possible implementations of the first aspect.
[0021] The load adjustment method, apparatus, device, and storage medium for the log collection system provided in this application determine the load status of the elastic search cluster by acquiring its pressure index information, thereby adjusting the current batch size parameter of the log collection processing component to obtain the target batch size parameter. Based on the target batch size parameter, the configuration file of the log collection processing component is updated, enabling the component to perform log collection for the elastic search cluster according to the updated configuration file. Compared with existing technologies, this method achieves adaptive rate limiting in the log collection system, avoiding log loss or service interruption caused by elastic search cluster overload, and improving the stability of the log collection system. Attached Figure Description
[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0023] Figure 1 This application provides a schematic diagram of a load adjustment system architecture for a log collection system.
[0024] Figure 2A flowchart illustrating a load adjustment method for a log collection system provided in this application. Figure 1 ;
[0025] Figure 3 A flowchart illustrating a load adjustment method for a log collection system provided in this application. Figure 2 ;
[0026] Figure 4 A schematic diagram of the structure of a load adjustment device for a log acquisition system provided in this application;
[0027] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.
[0028] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.
[0031] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.
[0032] It should be noted that the load adjustment method, apparatus, equipment and storage medium of the log collection system provided in this application can be used in the field of big data, or in any field other than big data. The application field of the load adjustment method, apparatus, equipment and storage medium of the log collection system in this application is not limited.
[0033] Existing log collection systems typically use statically configured log collector parameters to control the rate at which data is written to the Elastic Search Cluster. However, with the deepening of enterprise IT infrastructure development, the number and scale of application systems that need to be connected to the log center are growing exponentially, and the amount of log data is also increasing dramatically. This causes the Elastic Search Cluster to be overloaded, increasing the pressure on the CPU, virtual machine garbage collection mechanism, and indexing threads within the Elastic Search Cluster, resulting in write delays, request rejections, and node instability.
[0034] To avoid stability issues in the log collection system, it is usually necessary to monitor the load of the Elastic Search cluster during peak log collection periods, and then manually modify the log collector parameters in the static configuration file and restart the Elastic Search cluster.
[0035] However, in the existing technology, relying on manual modification of log collector parameters in static configuration files makes it difficult to respond to load changes in a timely manner. The existing technology suffers from insufficient real-time performance and high operation and maintenance costs, and cannot guarantee the stability of the log collection system.
[0036] To address the aforementioned issues, the core concept of this application is as follows: By collecting real-time pressure metrics of the elastic search cluster in the log collection system, the load status of the elastic search cluster is determined, thereby adjusting the current batch size parameter of the log collection processing component in the log collection system to obtain the target batch size parameter; based on the target batch size parameter, the configuration file of the log collection processing component is updated, enabling the log collection processing component to perform log collection for the elastic search cluster according to the updated configuration file. This achieves adaptive rate limiting in the log collection system, avoiding the problem of manual operation and maintenance failing to respond to load changes in a timely manner, thus improving the stability of the log collection system.
[0037] Optionally, Figure 1 This is a schematic diagram of a load conditioning system architecture for a log collection system provided in this application. Figure 1 As shown, the load adjustment system architecture of the log collection system includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.
[0038] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the above architecture. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.
[0039] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface. The data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface. The data acquisition device 101 can be used to obtain the pressure index information of the elastic search cluster in the log collection system.
[0040] The processing device 102 can be used to determine the load status of the elastic search cluster based on stress index information; adjust the current batch size parameter of the log collection and processing component in the log collection system according to the load status to obtain the target batch size parameter; and update the configuration file of the log collection and processing component according to the target batch size parameter so that the log collection and processing component can perform log collection for the elastic search cluster according to the updated configuration file.
[0041] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.
[0042] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0043] Figure 2 A flowchart illustrating a load adjustment method for a log collection system provided in this application. Figure 1 ,like Figure 2 As shown, the method includes:
[0044] S201. Obtain pressure index information of the elastic search cluster in the log collection system.
[0045] In this embodiment, the log collection system is used to collect, transmit, and write log data to the elastic search cluster. The elastic search cluster is a cluster composed of multiple elastic search nodes, used to store, retrieve, and analyze log data. Stress metrics are quantitative indicators reflecting the operating load of the elastic search cluster, including CPU utilization, memory utilization, or thread pool blocking rate.
[0046] S202. Determine the load status of the elastic search cluster based on the stress index information.
[0047] In this embodiment, the load status is the load status of the elastic search cluster determined by comparing the pressure index information with a preset load judgment threshold. The load status includes light load status, medium load status, or heavy load status.
[0048] S203. Based on the load status, adjust the current batch size parameter of the log collection and processing component in the log collection system to obtain the target batch size parameter.
[0049] In this embodiment, the log collection and processing component is the core component responsible for collecting, filtering, and pushing log data to the elastic search cluster. The batch size parameter is the amount of log data pushed to the elastic search cluster each time by the log collection and processing component. The current batch size parameter is the batch size parameter currently in effect for the log collection and processing component. The target batch size parameter is the new batch size parameter obtained by adjusting the current batch size parameter based on the load status.
[0050] S204. Update the configuration file of the log collection and processing component according to the target batch size parameter, so that the log collection and processing component can perform log collection for the Elastic Search cluster according to the updated configuration file.
[0051] In this embodiment, the configuration file is the main configuration file of the log collection and processing component, which is used to store batch size parameters.
[0052] For example, the log collection system automatically modifies the corresponding batch size parameter in the configuration file based on the target batch size parameter, triggering a smooth restart of the log collection and processing component. This allows the updated configuration file of the log collection and processing component to take effect, enabling the log collection and processing component to push log data to the elastic search cluster based on the target batch size parameter.
[0053] In some embodiments, steps S201 to S204 are executed cyclically to form an adaptive control closed loop between the load of the elastic search cluster and the consumption rate of the log collection and processing component; to ensure that the log collection system runs stably during peak periods and recovers automatically; and to realize a fully unattended adaptive tuning mechanism, thereby reducing manual operation and maintenance costs.
[0054] This application provides a load adjustment method for a log collection system. By acquiring stress indicator information of the Elastic Search Cluster (ESC), it comprehensively and accurately quantifies the ESC's operational load, providing a reliable data foundation for subsequent load status determination. The stress indicator information includes CPU utilization, memory utilization, or thread pool blocking rate, avoiding biased load status judgment due to a single stress indicator. By identifying light, medium, and heavy load states of the ESC based on stress indicator information, the current operational stress level of the ESC is clearly defined, providing a clear decision-making basis for subsequent batch size parameter adjustments. Dynamically adjusting the batch size parameter according to the load status achieves adaptive control of the log collection system. This reduces the batch size parameter to alleviate ESC pressure under high load and increases the batch size parameter to improve throughput under low load, effectively balancing the stability and processing efficiency of the log collection system. Automatically updating the configuration file and triggering a smooth restart ensures the timely application of the target batch size parameter, achieving dynamic adjustment of the log collection rate and guaranteeing the stability of the ESC under high load and resource utilization under low load.
[0055] Figure 3 A flowchart illustrating a load adjustment method for a log collection system provided in this application. Figure 2 ,exist Figure 2 Based on the illustrated embodiment, the load adjustment method of the log collection system will be described in detail, such as... Figure 3 As shown, the method includes:
[0056] S301. Periodically call the application programming interface of the Elastic Search Cluster to collect the CPU utilization of the Elastic Search Cluster through the application programming interface.
[0057] The application programming interface (API) of the Elastic Search Cluster is a node interface, used to obtain CPU utilization, memory utilization, or thread pool blocking rate within the Elastic Search Cluster. CPU utilization is a core indicator of the current operating pressure of the Elastic Search Cluster, directly reflecting the load on the cluster's log write requests. Therefore, in this embodiment, CPU utilization is used as a pressure indicator.
[0058] For example, according to the preset collection frequency, the CPU utilization of the elastic search cluster is obtained through the node interface, and the CPU utilization is used as a stress indicator. The preset collection frequency can be 1 minute, 5 minutes or 10 minutes, and the preset collection frequency can be dynamically adjusted according to the operation and needs of the log collection system.
[0059] The load adjustment method for the log collection system provided in this application obtains the CPU utilization of the Elastic Search Cluster as a stress indicator by periodically calling the node interface. This accurately and timely quantifies the load status of the log collection system, providing accurate supporting data for subsequent dynamic adjustment of batch size parameters.
[0060] S302. Obtain the load judgment threshold corresponding to the pressure index information; the load judgment threshold includes a first judgment threshold, a second judgment threshold and a third judgment threshold, the first judgment threshold is greater than the second judgment threshold, and the second judgment threshold is greater than the third judgment threshold.
[0061] The stress indicator information is the CPU utilization rate of the Elastic Search Cluster collected in step S301 above. The load judgment threshold is a predefined load threshold for multi-level CPUs in the log collection system, used to classify the stress state of the Elastic Search Cluster. The load judgment threshold can be flexibly adjusted according to the hardware configuration of the Elastic Search Cluster and the amount of log data written; among them, the first judgment threshold is the heavy load threshold, the second judgment threshold is the medium load threshold, and the third judgment threshold is the light load threshold.
[0062] For example, the log collection system loads the load judgment thresholds corresponding to the pressure index information from the configuration file: the first judgment threshold is set to 85%, the second judgment threshold is set to 70%, and the third judgment threshold is set to 40%.
[0063] S303. If the pressure index information is greater than the first judgment threshold, the load state is determined to be a heavy load state.
[0064] For example, if the CPU utilization of the Elastic Search Cluster is detected to be greater than 85%, it indicates that the Elastic Search Cluster is in a state of high load or even overload risk. Therefore, the load state is determined to be a state of heavy load.
[0065] S304. If the pressure index information is not greater than the first judgment threshold but is greater than the second judgment threshold, then the load status is determined to be a medium load status.
[0066] For example, if the CPU utilization of the Elastic Search Cluster is detected to be no more than 85% and greater than 70%, it indicates that the Elastic Search Cluster is beginning to experience significant pressure, but has not yet reached the risk of overload. Therefore, the load status is determined to be a moderate load status.
[0067] S305. If the pressure index information is less than the third judgment threshold, the load state is determined to be a light load state.
[0068] For example, if the CPU utilization of the Elastic Search Cluster is less than 40%, it indicates that the resources of the Elastic Search Cluster are relatively idle and underutilized. Therefore, the load status is determined to be a light load status.
[0069] The load adjustment method for the log collection system provided in this application establishes a quantitative correspondence between the utilization rate of the central processing unit and the load status of the elastic search cluster by setting a pre-defined hierarchical load judgment threshold, clarifies the judgment criteria for different load statuses, and improves the accuracy of subsequent load status identification.
[0070] S306. Determine how to adjust the batch size parameter of the log collection and processing component based on the load status.
[0071] The adjustment methods include increasing or decreasing the data volume.
[0072] In some embodiments, determining the adjustment method for the batch size parameter of the log collection and processing component based on the load status includes:
[0073] If the load status is heavy load, the adjustment method is to reduce the first data volume.
[0074] For example, the first data volume can be 50%. If the load status is heavy load, the adjustment method is to reduce it by 50%.
[0075] If the load status is moderate, the adjustment method is to reduce the second data volume; where the second data volume is less than the first data volume.
[0076] For example, the second data volume can be 20%. If the load status is medium load, the adjustment method is to reduce it by 20%.
[0077] If the load status is light load, the adjustment method is to increase the third data volume.
[0078] For example, the third data volume can be 20%. If the load status is light load, the adjustment method is to increase it by 20%.
[0079] The load adjustment method for the log collection system provided in this application ensures that the adjustment strategy can specifically alleviate the pressure on the elastic search cluster or improve the resource utilization of the elastic search cluster by quantitatively binding the load status with the adjustment method, and provides a clear execution direction for adjusting the batch size parameter.
[0080] S307. Adjust the current batch size parameter of the log collection and processing component according to the adjustment method and the preset effective range constraints to obtain the target batch size parameter.
[0081] The preset effective range constraints are the value boundaries of the batch size parameter, including the minimum batch size parameter and the maximum batch size parameter. The minimum batch size parameter is used to prevent the batch size parameter from being too small, which would lead to excessive latency in a single processing run; the maximum batch size parameter is used to prevent the batch size parameter from being too large, which would lead to memory overflow. The target batch size parameter is the batch size parameter that, after adjustment, still conforms to the preset effective range constraints.
[0082] For example, the minimum batch size parameter can be 125 or 250, and the maximum batch size parameter can be 2000 or 3000; the current batch size parameter is 1000, and the effective range constraint can be between 250 and 2000; if the load condition is heavy load, the target batch size parameter calculated by adjusting by 50% is 500, which meets the effective range constraint; if the load condition is medium load, the target batch size parameter calculated by adjusting by 20% is 800, which meets the effective range constraint; if the load condition is light load, the target batch size parameter calculated by adjusting by 20% is 1200, which meets the effective range constraint.
[0083] In some embodiments, if the current batch size parameter is 6000 and the load status is heavy load, the target batch size parameter calculated by adjusting by 50% is 3000, which does not meet the effective range constraint. Therefore, the maximum batch size parameter is used as the target batch size parameter.
[0084] The load adjustment method for the log collection system provided in this application determines the adjustment method for the batch size parameter of the log collection and processing component based on the load status. It precisely binds light, medium, and heavy load states with differentiated adjustment methods to ensure that the adjustment strategy matches the pressure of the elastic search cluster in real time. Based on the adjustment method and preset effective range constraints, the current batch size parameter of the log collection and processing component is adjusted to obtain the target batch size parameter. The target batch size parameter is dynamically calculated in conjunction with the effective range constraints, thereby reducing batch size to alleviate pressure under high load and increasing batch size to utilize resources under low load, while avoiding the risk of abnormal batch size parameters, achieving accuracy, security, and adaptability in load adjustment.
[0085] In some embodiments, before adjusting the current batch size parameter of the log collection and processing component in the log collection system according to the load status to obtain the target batch size parameter, the method further includes:
[0086] S308. Store the current batch size parameters to the configuration backup file.
[0087] In this embodiment, the configuration backup file is a copy of the main configuration file of the log collection and processing component, used to record historical configuration status.
[0088] For example, before modifying the configuration file of the log collection and processing component and adjusting the current batch size parameter, the log collection system completely copies the current configuration file into a configuration backup file, and notes the current batch size parameter and modification time in the configuration backup file; if the effect of subsequent adjustment of the target batch size parameter is not good or an anomaly occurs, the configuration backup file can be used to quickly roll back to the stable configuration before the adjustment.
[0089] The load adjustment method for the log collection system provided in this application establishes a configuration rollback mechanism by backing up the current batch size parameter to the configuration backup file, thereby reducing the risk of batch size parameter adjustment and ensuring the reliability of the log collection system.
[0090] In some embodiments, after updating the configuration file of the log collection and processing component according to the target batch size parameter, the method further includes:
[0091] S309. Restart the log collection and processing component to trigger log collection for the Elastic Search cluster.
[0092] In this embodiment, the log collection and processing component is restarted by executing a service management command, so that the updated configuration file of the log collection and processing component takes effect, thereby triggering log collection for the Elastic Search cluster and ensuring the stable operation of the log collection system.
[0093] In some embodiments, restarting the log collection and processing component includes:
[0094] Check the component status of the log collection and processing component.
[0095] In this embodiment, the log collection system detects whether the queue type configuration in the configuration file of the log collection and processing component is in persistent mode, and verifies the validity of the corresponding queue path to ensure that the component status of the log collection and processing component is that the persistent queue function has been started.
[0096] If the component status is "Persistent queue function started", then restart the log collection and processing component.
[0097] The persistent queue function writes unprocessed log data to disk, and resumes processing the log data from the point of interruption after the log collection and processing component is restarted, thus avoiding data loss during the restart process.
[0098] In some embodiments, if the log collection and processing component is deployed in a multi-instance elastic search cluster environment, a rolling restart strategy can be adopted to restart the log collection and processing component one by one or in batches to ensure that the elastic search cluster is online to process log data at any time, avoid service interruption and log processing pause, and improve the continuity and stability of the log collection system.
[0099] The load adjustment method for the log collection system provided in this application ensures that the target batch size parameter takes effect while avoiding log data loss by enabling the restart operation after the persistent queue is enabled; the rolling restart strategy in a multi-instance elastic search cluster environment reduces the risk of service interruption and ensures the continuity of log collection.
[0100] Figure 4 A schematic diagram of the structure of a load adjustment device for a log acquisition system provided in this application is shown below. Figure 4 As shown, the load adjustment device of the log collection system includes:
[0101] The acquisition module 401 is used to acquire pressure index information of the elastic search cluster in the log collection system.
[0102] The determination module 402 is used to determine the load status of the elastic search cluster based on the stress index information.
[0103] The adjustment module 403 is used to adjust the current batch size parameter of the log collection and processing component in the log collection system according to the load status, so as to obtain the target batch size parameter.
[0104] Update module 404 is used to update the configuration file of the log collection and processing component according to the target batch size parameter, so that the log collection and processing component can perform log collection for the Elastic Search cluster according to the updated configuration file.
[0105] In one possible implementation, the stress metric is the CPU utilization of the elastic search cluster; correspondingly, the acquisition module 401 can also be used for:
[0106] Accordingly, obtain the stress metrics information of the elastic search cluster in the log collection system, including:
[0107] Periodically call the Elastic Search Cluster's application programming interface (API) to collect the Elastic Search Cluster's CPU utilization through the API.
[0108] In one possible implementation, the determining module 402 can also be used for:
[0109] Obtain the load judgment threshold corresponding to the pressure index information; the load judgment threshold includes a first judgment threshold, a second judgment threshold and a third judgment threshold, the first judgment threshold is greater than the second judgment threshold, and the second judgment threshold is greater than the third judgment threshold;
[0110] If the pressure index information is greater than the first judgment threshold, the load status is determined to be a heavy load status;
[0111] If the pressure index information is not greater than the first judgment threshold but greater than the second judgment threshold, then the load status is determined to be a medium load status.
[0112] If the pressure index information is less than the third judgment threshold, the load status is determined to be a light load status.
[0113] In one possible implementation, the adjustment module 403 can also be used for:
[0114] Determine how to adjust the batch size parameter of the log collection and processing component based on the load status;
[0115] Based on the adjustment method and preset effective range constraints, adjust the current batch size parameter of the log collection and processing component to obtain the target batch size parameter.
[0116] In one possible implementation, the adjustment module 403 can also be used for:
[0117] If the load status is heavy load, the adjustment method is to reduce the first data volume;
[0118] If the load status is moderate, the adjustment method is to reduce the second data volume; where the second data volume is less than the first data volume.
[0119] If the load status is light load, the adjustment method is to increase the third data volume.
[0120] In one possible implementation, before adjusting the current batch size parameter of the log collection and processing component in the log collection system according to the load status to obtain the target batch size parameter, the load adjustment device of the log collection system further includes a storage module, which can specifically be used for:
[0121] Store the current batch size parameter to the configuration backup file.
[0122] In one possible implementation, after updating the configuration file of the log collection and processing component according to the target batch size parameter, the load adjustment device of the log collection system also includes a restart module, which can specifically be used for:
[0123] Restart the log collection and processing component to trigger log collection for the Elastic Search cluster.
[0124] In one possible implementation, the restart module can also be used for:
[0125] Detect the component status of the log collection and processing component;
[0126] If the component status is "Persistent queue function started", then restart the log collection and processing component.
[0127] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0128] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0129] Figure 5 A schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5 As shown, the electronic device includes at least one processor 501 and a memory 502. Optionally, the electronic device also includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0130] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.
[0131] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0132] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0133] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0134] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0135] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0136] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0137] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0138] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0139] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0140] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0141] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0142] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as a portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0143] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0144] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A load adjustment method for a log collection system, characterized in that, include: Obtain stress metrics information for the elastic search cluster in the log collection system; Based on the stress index information, determine the load status of the elastic search cluster; Based on the load status, adjust the current batch size parameter of the log collection and processing component in the log collection system to obtain the target batch size parameter; The configuration file of the log collection and processing component is updated according to the target batch size parameter, so that the log collection and processing component performs log collection for the elastic search cluster according to the updated configuration file.
2. The method according to claim 1, characterized in that, The step of adjusting the current batch size parameter of the log collection and processing component in the log collection system according to the load status to obtain the target batch size parameter includes: Based on the load status, determine how to adjust the batch size parameter of the log collection and processing component; Based on the adjustment method and the preset effective range constraints, the current batch size parameter of the log collection and processing component is adjusted to obtain the target batch size parameter.
3. The method according to claim 2, characterized in that, Determining the load status of the elastic search cluster based on the stress index information includes: Obtain the load judgment threshold corresponding to the pressure index information; the load judgment threshold includes a first judgment threshold, a second judgment threshold and a third judgment threshold, wherein the first judgment threshold is greater than the second judgment threshold and the second judgment threshold is greater than the third judgment threshold; If the pressure index information is greater than the first judgment threshold, then the load state is determined to be a heavy load state; If the pressure index information is not greater than the first judgment threshold but is greater than the second judgment threshold, then the load state is determined to be a medium load state. If the pressure index information is less than the third judgment threshold, then the load state is determined to be a light load state.
4. The method according to claim 3, characterized in that, The method for determining the adjustment of the batch size parameter of the log collection and processing component based on the load status includes: If the load state is the heavy load state, then the adjustment method is determined to be to reduce the first data volume; If the load state is the moderate load state, then the adjustment method is determined to be to reduce the second data volume; wherein the second data volume is less than the first data volume; If the load state is the light load state, then the adjustment method is determined to be to increase the third data volume.
5. The method according to any one of claims 1 to 4, characterized in that, The stress indicator information is the CPU utilization rate of the elastic search cluster; Accordingly, obtaining the stress indicator information of the elastic search cluster in the log collection system includes: The application programming interface (API) of the elastic search cluster is periodically invoked to collect the CPU utilization of the elastic search cluster through the API.
6. The method according to any one of claims 1 to 4, characterized in that, After updating the configuration file of the log collection and processing component according to the target batch size parameter, the method further includes: Restart the log collection and processing component to trigger log collection for the Elastic Search cluster.
7. The method according to claim 6, characterized in that, Restarting the log collection and processing component includes: Detect the component status of the log collection and processing component; If the component status is "persistent queue function enabled", then restart the log collection and processing component.
8. The method according to any one of claims 1 to 4, characterized in that, Before adjusting the current batch size parameter of the log collection and processing component in the log collection system according to the load status to obtain the target batch size parameter, the method further includes: Store the current batch size parameter to the configuration backup file.
9. A load adjustment device for a log acquisition system, characterized in that, include: The acquisition module is used to acquire stress indicator information of the elastic search cluster in the log collection system. The determination module is used to determine the load status of the elastic search cluster based on the pressure index information. An adjustment module is used to adjust the current batch size parameter of the log collection and processing component in the log collection system according to the load status, so as to obtain the target batch size parameter. An update module is used to update the configuration file of the log collection and processing component according to the target batch size parameter, so that the log collection and processing component performs log collection for the elastic search cluster according to the updated configuration file.
10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.