A method and system for dynamically adjusting slow log collection

Abstract

The present invention relates to the field of electronic digital processing technology, and in particular to a method and system for dynamically adjusting slow log collection, the method comprising the following steps: deploying a slow log execution method script; starting all slow log collectors and registering them to an operation and maintenance platform; the operation and maintenance platform calculating in real time the number of reasonable data collected by the slow log collector per minute, and comparing the sum with the number of data to be collected currently, and then using different task allocation methods to allocate the current data to be collected to the slow log collector according to the comparison result; the slow log collector collects and saves the corresponding data after receiving the collection task allocated by the operation and maintenance platform. The method provided by the present invention can dynamically adjust and allocate the tasks to be collected in real time, thereby improving the collection efficiency and reliability of the slow log.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital processing technology, and in particular to a method and system for dynamically adjusting slow log collection. Background Art

[0002] In the context of cloud computing and big data, database applications in public cloud environments are becoming more and more widespread. As the database runs, the amount of database business access increases, and the amount of database data increases, sometimes there is a situation where query results are returned slowly, so it is necessary to record slow query logs to find slow query statements that form bottlenecks. Databases are suitable for application scenarios of online processing systems, including key core business systems in the financial and telecommunications industries. However, when collecting database slow logs, the operation and maintenance platform generally allocates collection tasks based on the inherent capabilities of the existing slow log collectors and in the inherent order of the slow log collectors. When allocating collection tasks, it is simply allocated according to the order of the collection tasks to be allocated. This method ignores the fact that the actual collection capacity of the slow log collector changes dynamically with the use status of the slow log collector, and ignores the priority sorting problem of the slow log collector and the priority sorting problem of the task. It may lead to a decrease in the final collection efficiency and reliability. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for dynamically adjusting slow log collection. The operation and maintenance platform can dynamically adjust in real time according to the situation of each slow log collector and the situation of the tasks to be collected to allocate the tasks to be collected, thereby improving the collection efficiency of slow logs and improving the reliability of slow log collection.

[0004] The present invention is achieved through the following technical solutions:

[0005] S1: Deploy an instance in a public cloud environment and deploy the slow log related execution method script on the instance node;

[0006] S2: Start all slow log collectors and register them to the operation and maintenance platform;

[0007] S3: The operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute in real time, and compares the current number of data to be collected with the sum of the reasonable number of data collected by all slow log collectors per minute. Then, based on the comparison result, different task allocation methods are used to allocate the current data to be collected to the slow log collector.

[0008] S4: After receiving the collection task assigned by the operation and maintenance platform, the slow log collector collects and saves the corresponding data.

[0009] Further, in step S3, the operation and maintenance platform uses different task allocation methods according to the comparison results to allocate the current data to be collected to the slow log collector in the following way:

[0010] If the current number of data to be collected is less than or equal to the total number of data reasonably collected per minute by all slow log collectors, the operation and maintenance platform will assign the tasks to be collected to the corresponding slow log collectors according to the priority sorting of all slow log collectors based on the calculated number of data reasonably collected per minute by the slow log collectors; if the current number of data to be collected is greater than the total number of data reasonably collected per minute by all slow log collectors, the operation and maintenance platform will assign the tasks to be collected to all slow log collectors according to the priority sorting of collection tasks based on the calculated number of data reasonably collected per minute by the slow log collectors.

[0011] Furthermore, in step S3, the operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute in real time according to formula (1):

[0012] (1);

[0013] in Indicates The number of reasonable data items collected per minute by a slow log collector. Indicates the current priority set by the user. The number of data entries executed by a slow log collector, Indicates the user-set The slow log collectors failed to execute but did not exceed Each slow log collector sets the number of data entries for the number of failures. Indicates The number of collection tasks that are not currently executed by the slow log collector. Indicates The minimum number of collection tasks set for a slow log collector. Indicates The priority of a slow log collector, Indicates The number of CPU cores actually used by a slow log collector. Indicates The total number of CPU cores of the slow log collector, Indicates The actual amount of memory used by a slow log collector. Indicates The total memory size of the slow log collector.

[0014] Furthermore, the operation and maintenance platform calculates the Slow log collector priority :

[0015] (2);

[0016] in: Indicates The remaining disk space of the slow log collector storage database path, Indicates The total disk space of the slow log collector storage database path, Indicates A slow log collector queries the time it takes to collect a batch of slow logs. Indicates the average time taken by all slow log collectors to query a batch of slow log collection. Indicates The size of the remaining swap partition of the slow log collector, Indicates The total size of the swap partition of the slow log collector, Indicates The length of the request queue initiated by a slow log collector, Indicates the average length of the request queue initiated by all slow log collectors.

[0017] Furthermore, the operation and maintenance platform calculates the Priority of each collection task :

[0018] (3);

[0019] in: Indicates The space occupied by each collection task, Indicates the average space occupied by all tasks to be collected. Indicates The active duration of each collection task within the set time period, Indicates the duration of the set time period. Indicates The number of successful data collection of the same type in each collection task, Indicates the total number of successful data collections of the same type as all tasks to be collected.

[0020] Furthermore, the slow log collector periodically deletes slow logs whose storage time is longer than the set time.

[0021] Optimized, set the duration to 12 hours.

[0022] Optimized, the operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute at a frequency of 1 time per minute.

[0023] A slow log collection dynamic adjustment operation and maintenance platform comprises a task receiving module, a data processing module and a task distribution module. The task receiving module is connected with a user end and is used to receive tasks sent by the user end. The data processing module is used to calculate in real time the number of reasonable data collected by all slow log collectors per minute, and compare the current number of data to be collected with the sum of the number of reasonable data collected by all slow log collectors per minute. The task distribution module distributes tasks to corresponding slow log collectors according to the comparison result of the data processing module.

[0024] A slow log collection dynamic adjustment system is used to execute any one of the above-mentioned slow log collection dynamic adjustment methods, which includes a user end, a communication unit, an operation and maintenance platform and multiple slow log collectors. The user end is used for users to input tasks to be collected and transmit them to the operation and maintenance platform through the communication unit. The operation and maintenance platform includes a task receiving module, a data processing module and a task distribution module. The task receiving module is used to receive tasks sent by the user end. The data processing module is used to calculate in real time the number of reasonable data collected per minute by all slow log collectors, and compare the current number of data to be collected with the sum of the number of reasonable data collected per minute by all slow log collectors. The task distribution module distributes the tasks to the corresponding slow log collectors according to the comparison results of the data processing module and through the communication unit.

[0025] Beneficial effects of the invention:

[0026] The present invention provides a method and system for dynamically adjusting slow log collection. An operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute in real time, and compares the current number of data to be collected with the sum of the reasonable number of data collected by all slow log collectors per minute. If the current number of data to be collected is less than or equal to the sum of the reasonable number of data collected by all slow log collectors per minute, the task to be collected is assigned to the corresponding slow log collector according to the priority sorting of all slow log collectors. If the current number of data to be collected is greater than the sum of the reasonable number of data collected by all slow log collectors per minute, the operation and maintenance platform assigns the task to be collected to all slow log collectors according to the priority sorting of the collection tasks, thereby realizing real-time dynamic adjustment of slow log collection, giving full play to the advantages of each slow log collector, realizing priority collection of important collection tasks, not only ensuring the collection efficiency of slow log collection, but also effectively reducing the probability of failure of slow log collector to collect data, and improving the reliability of slow log collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0028] Figure 2 It is a schematic diagram of the system of the present invention.

[0029] In the figure: 1. User end; 2. Operation and maintenance platform; 21. Task receiving module; 22. Data processing module; 23. Task distribution module; 3. Slow log collector; 4. Communication unit. DETAILED DESCRIPTION

[0030] A method for dynamically adjusting slow log collection, the overall process diagram is as follows: Figure 1 As shown, the specific steps include:

[0031] S1: Deploy an instance in a public cloud environment and deploy the slow log related execution method script on the instance node;

[0032] S2: Start all slow log collectors and register them to the operation and maintenance platform;

[0033] In a public cloud environment, you can use the Java -jar command to start the corresponding slow log collector. After the slow log collector is started, it will access the registration log collector interface of the operation and maintenance platform through HTTP requests to register the current slow log collector on the operation and maintenance platform.

[0034] S3: The operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute in real time, and compares the current number of data to be collected with the sum of the reasonable number of data collected by all slow log collectors per minute. Then, based on the comparison result, different task allocation methods are used to allocate the current data to be collected to the slow log collector.

[0035] The operation and maintenance platform can calculate the number of reasonable data collected by all slow log collectors per minute once per minute.

[0036] Furthermore, the data to be collected can be allocated to the slow log collector using different task allocation methods according to the comparison results as follows:

[0037] If the current number of data to be collected is less than or equal to the total number of data reasonably collected per minute by all slow log collectors, the operation and maintenance platform will assign the tasks to be collected to the corresponding slow log collectors according to the priority sorting of all slow log collectors based on the calculated number of data reasonably collected per minute by the slow log collectors; if the current number of data to be collected is greater than the total number of data reasonably collected per minute by all slow log collectors, the operation and maintenance platform will assign the tasks to be collected to all slow log collectors according to the priority sorting of collection tasks based on the calculated number of data reasonably collected per minute by the slow log collectors.

[0038] Specifically, the operation and maintenance platform can calculate the number of reasonable data collected by all slow log collectors per minute in real time according to formula (1):

[0039] (1);

[0040] in Indicates The number of reasonable data items collected per minute by a slow log collector. Indicates the current priority set by the user. The number of data entries executed by a slow log collector, Indicates the user-set The slow log collectors failed to execute but did not exceed Each slow log collector sets the number of data entries for the number of failures. Indicates The number of collection tasks that are not currently executed by the slow log collector. Indicates The minimum number of collection tasks set for a slow log collector. Indicates The priority of a slow log collector, Indicates The number of CPU cores actually used by a slow log collector. Indicates The total number of CPU cores of the slow log collector, Indicates The actual amount of memory used by a slow log collector. Indicates The total memory size of the slow log collector.

[0041] The operation and maintenance platform uses the above method to calculate in real time the number of reasonable data collected by each slow log collector per minute, fully considering multiple factors affecting the collection capacity of each slow log collector, rather than simply allocating collection tasks based on the fixed collection capacity of each slow log collector, which can improve the reliability of collection tasks.

[0042] Furthermore, the operation and maintenance platform calculates the Slow log collector priority :

[0043] (2);

[0044] in: Indicates The remaining disk space of the slow log collector storage database path, Indicates The total disk space of the slow log collector storage database path, Indicates A slow log collector queries the time it takes to collect a batch of slow logs. Indicates the average time taken by all slow log collectors to query a batch of slow log collections. Indicates The size of the remaining swap partition for the slow log collector. Indicates The total size of the swap partition of the slow log collector, Indicates The length of the request queue initiated by a slow log collector, Indicates the average length of the request queue initiated by all slow log collectors.

[0045] By calculating the priority of the slow log collector according to the above method, the processing capacity of the slow log collector can be fully considered. When all the slow log collectors have insufficient tasks, the tasks to be assigned can be assigned to the slow log collectors with higher priorities in order of priority.

[0046] Furthermore, the operation and maintenance platform calculates the Priority of each collection task :

[0047] (3);

[0048] in: Indicates The space occupied by each collection task, Indicates the average space occupied by all tasks to be collected. Indicates The active duration of each collection task within the set time period, Indicates the duration of the set time period. Indicates The number of successful data collection of the same type in each collection task, Indicates the total number of successful data collections of the same type as all tasks to be collected.

[0049] The priority of the collection task is calculated according to the above method, which fully considers factors such as the size, importance and probability of success of the task. When the collection task is greater than the processing capacity, the collection task with the higher priority can be allocated first.

[0050] The present invention provides a dynamic adjustment method for slow log collection. An operation and maintenance platform calculates in real time the number of data reasonably collected by all slow log collectors per minute, and then compares the current number of data to be collected with the sum of the number of data reasonably collected by all slow log collectors per minute. If the current number of data to be collected is less than or equal to the sum of the number of data reasonably collected by all slow log collectors per minute, it indicates that the collection capacity of the slow log collector is sufficient. At this time, combined with the number of data reasonably collected by the slow log collector per minute calculated in real time, the tasks to be collected are assigned to the corresponding slow log collectors according to the priority sorting of all slow log collectors.

[0051] The specific instructions are as follows:

[0052] Assume that the number of task data to be processed is 1,000 and the number of slow log collectors is three. The number of data that the slow log collector can reasonably collect per minute is: 450 460 There are 480 records, and the priority order calculated by the three slow log collectors is > > , when the operation and maintenance platform allocates collection tasks, it first allocates 460 data to the second slow log collector, then allocates 450 data to the first slow log collector, and then allocates the remaining 90 data to the third slow log collector.

[0053] In this way, when the processing capacity of the slow log collector is sufficient, the real-time processing capacity of each slow log collector is reasonably considered, and the collection tasks are allocated according to the priority order according to the specific situation of each slow log collector, which can improve the collection efficiency and reliability.

[0054] If the number of data to be collected is greater than the total number of data reasonably collected by all slow log collectors per minute, it means that the collection capacity of the slow log collector is insufficient. In this case, the operation and maintenance platform will assign the tasks to be collected to all slow log collectors according to the collection task priority.

[0055] The specific instructions are as follows:

[0056] Assuming that the number of task data to be processed is 1500, and the number of slow log collectors is still three, the number of data reasonably collected per minute by the slow log collector for real-time calculation is still: 450 460 There are 480 records, and the priority order calculated by the three slow log collectors is > > , then when the operation and maintenance platform allocates collection tasks, it can first calculate the priority of each collection task according to formula (3) Then, according to the priority order of the collection tasks, 450 items are allocated to the first slow log collector, 460 items to the second slow log collector, and 480 items to the third slow log collector. Then, the remaining collection tasks can be allocated to the slow log collector that is processed first or allocated according to the priority order of the slow log collectors.

[0057] The above calculation process is automatically completed by the operation and maintenance platform, and the calculation speed is much faster than the waiting time of the slow log collector. By adopting the above method, the risks brought by the operation and maintenance personnel directly operating the database to modify the configuration can be reduced. Through the operation and maintenance platform to calculate and dynamically allocate slow log collection tasks, the advantages of each slow log collector can be fully utilized, and the performance of the machine can be maximized to collect slow logs, and important collection tasks can be collected first, which not only ensures the collection efficiency of slow log collection, but also effectively reduces the probability of failure of slow log collectors to collect data, improves the processing capacity of key information, and increases the reliability of slow log collection.

[0058] S4: After receiving the collection task assigned by the operation and maintenance platform, the slow log collector collects and saves the corresponding data.

[0059] Furthermore, the slow log collector periodically deletes slow logs whose storage time is longer than a set time, and the preferred set time is 12 hours. This can effectively further improve the dynamic collection capability of the slow log collector.

[0060] A slow log collection dynamic adjustment operation and maintenance platform comprises a task receiving module, a data processing module and a task distribution module. The task receiving module is connected with a user end and is used to receive tasks sent by the user end. The data processing module is used to calculate in real time the number of reasonable data collected by all slow log collectors per minute, and compare the current number of data to be collected with the sum of the number of reasonable data collected by all slow log collectors per minute. The task distribution module distributes tasks to corresponding slow log collectors according to the comparison result of the data processing module.

[0061] A dynamic adjustment system for slow log collection, as shown in the system diagram Figure 2As shown, a method for dynamically adjusting slow log collection for executing any one of the above items includes a user terminal 1, a communication unit 4, an operation and maintenance platform 2 and multiple slow log collectors 3, wherein the user terminal is used for the user to input the tasks to be collected and transmit them to the operation and maintenance platform through the communication unit, and the operation and maintenance platform includes a task receiving module 21, a data processing module 22 and a task distribution module 23, wherein the task receiving module is used to receive the tasks sent by the user terminal, the data processing module is used to calculate in real time the number of reasonable data collected per minute by all slow log collectors, and compare the current number of data to be collected with the sum of the number of reasonable data collected per minute by all slow log collectors, and the task distribution module distributes the tasks to the corresponding slow log collectors according to the comparison results of the data processing module and through the communication unit.

[0062] In summary, the present invention provides a method and system for dynamic adjustment of slow log collection, which realizes real-time dynamic adjustment of slow log collection, can give full play to the advantages of each slow log collector, and realizes priority collection of important collection tasks, which not only ensures the collection efficiency of slow log collection, but also effectively reduces the probability of failure of slow log collector to collect data, and improves the reliability of slow log collection.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for dynamically adjusting slow log collection, characterized in that: The steps include: S1: Deploy an instance in a public cloud environment and deploy the slow log related execution method script on the instance node; S2: Start all slow log collectors and register them to the operation and maintenance platform; S3: The operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute in real time according to formula (1), and compares the current number of data to be collected with the sum of the reasonable number of data collected by all slow log collectors per minute. Then, the operation and maintenance platform uses different task allocation methods according to the comparison results to allocate the current data to the slow log collector as follows: If the current number of data to be collected is less than or equal to the sum of the number of reasonable data collected by all slow log collectors per minute, the operation and maintenance platform will assign the tasks to be collected to the corresponding slow log collectors according to the priority sorting of all slow log collectors based on the calculated number of reasonable data collected by the slow log collectors per minute; if the current number of data to be collected is greater than the sum of the number of reasonable data collected by all slow log collectors per minute, the operation and maintenance platform will assign the tasks to be collected to all slow log collectors according to the priority sorting of the collection tasks based on the calculated number of reasonable data collected by the slow log collectors per minute; （1）； in Indicates The number of reasonable data items collected per minute by a slow log collector. Indicates the current priority set by the user. The number of data entries executed by the slow log collector. Indicates the user-set The slow log collectors failed to execute but did not exceed Each slow log collector sets the number of data entries for the number of failures. Indicates The number of collection tasks that are not currently executed by the slow log collector. Indicates The minimum number of collection tasks set for a slow log collector. Indicates The priority of a slow log collector, Indicates The number of CPU cores actually used by a slow log collector. Indicates The total number of CPU cores of the slow log collector, Indicates The actual amount of memory used by a slow log collector. Indicates Total memory of slow log collectors; S4: After receiving the collection task assigned by the operation and maintenance platform, the slow log collector collects and saves the corresponding data.

2. According to claim 1, a method for dynamically adjusting slow log collection is characterized in that: The operation and maintenance platform calculates the Slow log collector priority : （2）； in: Indicates The remaining disk space of the slow log collector storage database path, Indicates The total disk space of the slow log collector storage database path, Indicates A slow log collector queries the time it takes to collect a batch of slow logs. Indicates the average time taken by all slow log collectors to query a batch of slow log collection. Indicates The size of the remaining swap partition of the slow log collector, Indicates The total size of the swap partition of the slow log collector, Indicates The length of the request queue initiated by a slow log collector, Indicates the average length of the request queue initiated by all slow log collectors.

3. According to claim 1, a method for dynamically adjusting slow log collection is characterized in that: The operation and maintenance platform calculates the Priority of each collection task : （3）； in: Indicates The space occupied by each collection task, Indicates the average space occupied by all tasks to be collected. Indicates The active duration of each collection task within the set time period, Indicates the duration of the set time period. Indicates The number of successful data collection of the same type in each collection task, Indicates the total number of successful data collections of the same type as all tasks to be collected.

4. According to claim 1, a method for dynamically adjusting slow log collection is characterized in that: The slow log collector periodically deletes slow logs that have been saved for longer than the set time.

5. According to claim 4, a method for dynamically adjusting slow log collection is characterized in that: The set duration is 12 hours.

6. According to claim 1, a method for dynamically adjusting slow log collection is characterized in that: The operation and maintenance platform calculates the number of reasonable data collected by all slow log collectors per minute at a frequency of 1 time per minute.

7. A slow log collection dynamic adjustment operation and maintenance platform, used to execute a slow log collection dynamic adjustment method according to any one of claims 1 to 6, characterized in that: It includes a task receiving module, a data processing module and a task distribution module. The task receiving module is connected to the user end and is used to receive tasks sent by the user end. The data processing module is used to calculate in real time the number of reasonable data collected by all slow log collectors per minute, and compare the current number of data to be collected with the total number of reasonable data collected by all slow log collectors per minute. The task distribution module distributes the task to the corresponding slow log collector according to the comparison result of the data processing module.

8. A system for dynamically adjusting slow log collection, used to execute a method for dynamically adjusting slow log collection as claimed in any one of claims 1 to 6, characterized in that: The invention comprises a user terminal, a communication unit, an operation and maintenance platform and a plurality of slow log collectors. The user terminal is used for the user to input the tasks to be collected and transmit them to the operation and maintenance platform through the communication unit. The operation and maintenance platform comprises a task receiving module, a data processing module and a task distribution module. The task receiving module is used to receive the tasks sent by the user terminal. The data processing module is used to calculate in real time the number of reasonable data collected per minute by all slow log collectors, and compare the number of data to be collected currently with the sum of the number of reasonable data collected per minute by all slow log collectors. The task distribution module distributes the tasks to the corresponding slow log collectors according to the comparison result of the data processing module and through the communication unit.

Images