Streaming data collector resource automatic adjusting and optimizing method and device and computer equipment

Abstract

The invention discloses a streaming data collector resource automatic tuning method and device and computer equipment. The method comprises the steps that a table name list is received, and the number of current tables is calculated; starting an acquisition operation to read binlog data from an upstream database and write the binlog data into a downstream system, and obtaining a data processing rate, an input rate and a total accumulated number to obtain monitoring data; calculating an optimal table number according to the monitoring data and the current table number to obtain a calculation result; and readjusting the acquisition list according to the calculation result, and restarting the acquisition operation. By implementing the method provided by the invention, data processing delay can be effectively avoided, waste of resources of the collector can be prevented, and the optimal balance between resource utilization efficiency and system stability is realized; the data processing speed and accuracy can be improved, collector resources can be utilized to the maximum extent, and the operation efficiency of the whole system is improved.

Description

Automatic resource optimization method, apparatus and computer equipment for streaming data acquisition Technical Field

[0001] This invention relates to optimization methods, and more specifically to automatic optimization methods, apparatus, and computer equipment for streaming data acquisition resources. Background Technology

[0002] In the current technological framework, application systems typically need to transmit data to big data systems in real time to meet data analysis requirements. This data transmission process involves multiple upstream application systems, each with its own specific data structure and format, and requires data to be transmitted to corresponding data tables within the big data system. Due to the diversity and complexity of upstream application systems, and the sheer number of data tables to be processed, it is usually necessary to deploy a large number of data collectors to simultaneously collect and transmit data.

[0003] However, in practice, the data volume and size of each upstream data table often vary significantly. Using a fixed configuration to allocate a fixed number of data tables to each collector can lead to some tables having large data volumes or large data sets, resulting in severe delays. Conversely, when the data volume is small, the collector's resource utilization is very low, resulting in idle resources. This situation affects both the timeliness and speed of data processing, wastes system resources, and increases collection costs. When the data volume suddenly increases, collectors operating with a fixed configuration may experience severe delays due to their inability to process the excessive data in time; conversely, when the data volume is small, if the collector continues to operate with the original configuration, it may lead to low resource utilization, i.e., idle resources. This situation not only affects the efficiency and speed of data processing but also wastes system resources, thereby increasing operating costs.

[0004] Therefore, it is necessary to design a new method to ensure that data processing delays are effectively avoided while preventing the waste of the collector's own resources, and to achieve the best balance between resource utilization efficiency and system stability. This method can not only improve the speed and accuracy of data processing, but also maximize the utilization of collector resources and improve the overall system performance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus and computer equipment for automatic optimization of streaming data acquisition resources.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an automatic resource optimization method for streaming data acquisition machines, comprising:

[0007] Receive a list of table names to obtain the current number of tables;

[0008] Start the data collection job to read binlog data from the upstream database and write it to the downstream system. Obtain the data processing rate, input rate, and total backlog of records to obtain monitoring data.

[0009] The optimal number of tables is calculated based on the monitoring data and the current number of tables to obtain the calculation result;

[0010] The data collection list is readjusted based on the calculation results, and the data collection job is restarted to make the new data collection list effective in the data collection executor.

[0011] The further technical solution is as follows: the initiated data collection operation reads binlog data from the upstream database and writes it to the downstream system to obtain the data processing rate, input rate, and total backlog of records, in order to obtain monitoring data, including:

[0012] The data collection job is started periodically to read binlog data from the upstream database and write it to the downstream system. The data processing rate, input rate and total backlog number are obtained to obtain monitoring data.

[0013] The further technical solution is as follows: The step of calculating the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result includes:

[0014] Based on the monitoring data and the current number of tables, the optimal number of tables is calculated according to the set interval to optimize the collection efficiency and obtain the calculation result.

[0015] The further technical solution is as follows: The optimal number of tables is calculated based on the monitoring data and the current number of tables according to a set interval to optimize data collection efficiency, and the calculation result is obtained, including:

[0016] use The optimal number of tables is calculated to optimize data collection efficiency, resulting in the calculation results, where T... n ′ The optimal number of tables; Ins is the input rate; B is the total backlog; Er is the settling time; P s T represents the data processing rate. n This represents the current number of items in the table.

[0017] The further technical solution is as follows: the step of readjusting the data acquisition table list based on the calculation results and restarting the data acquisition operation includes:

[0018] The number of current tables may be increased or decreased based on the calculation results.

[0019] A further technical solution is as follows: increasing or decreasing the current table quantity based on the calculation result includes:

[0020] The calculation results are compared with the current table count to obtain a comparison result;

[0021] Increase or decrease the number of current tables based on the comparison results, and restart the data collection executor to make the changes take effect.

[0022] The further technical solution is as follows: Increasing or decreasing the current table count based on the comparison results and restarting the data collection executor to make the changes take effect includes:

[0023] If the difference between the two is greater than zero, add the corresponding number of new tables from the upstream database to the current table count; if the difference is less than zero, remove the corresponding number of tables from the current table count and restart the data collection executor to ensure that the adjusted configuration takes effect.

[0024] The present invention also provides an automatic resource optimization device for streaming data acquisition, comprising:

[0025] The data collection and execution unit is used to receive a list of table names and calculate the number of tables currently in use.

[0026] The monitoring unit is used to initiate a data acquisition operation to read binlog data from the upstream database and write it to the downstream system, and to obtain data processing rate, input rate and total backlog number to obtain monitoring data.

[0027] The calculation unit is used to calculate the optimal number of tables based on the monitoring data and the current number of tables, so as to obtain the calculation result;

[0028] The table allocation unit is used to readjust the collection table list based on the calculation results and restart the collection job.

[0029] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0030] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0031] The advantages of this invention compared to existing technologies are as follows: By dynamically adjusting the number of tables from which binlog data is read from the upstream database and written to the downstream system, this invention ensures high efficiency and accuracy in data processing. First, it receives a list of current table names to determine the current task configuration and acquires monitoring data, including data processing rate, input rate, and total backlog, to assess system performance and load. Then, based on this information, it calculates the optimal number of tables to effectively handle data backlog, avoid delays, and prevent waste caused by over-allocation of resources. Finally, it automatically adjusts the number of tables based on the calculation results, achieving the best balance between resource utilization efficiency and system stability. This not only improves the speed and accuracy of single data processing but also maximizes the utilization of the collector's resources, ultimately improving the overall system's operational efficiency and reliability.

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 is a schematic diagram of an application scenario of the automatic resource optimization method for streaming data collectors provided in an embodiment of the present invention.

[0035] Figure 2 is a flowchart illustrating the automatic resource optimization method for streaming data collectors provided in an embodiment of the present invention.

[0036] Figure 3 is a schematic block diagram of the automatic resource optimization device for streaming data acquisition provided in an embodiment of the present invention;

[0037] Figure 4 is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0040] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0041] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0042] Please refer to Figures 1 and 2. Figure 1 is a schematic diagram of an application scenario for the automatic resource optimization method for streaming data collectors provided in this embodiment of the invention. Figure 2 is a schematic flowchart of the automatic resource optimization method for streaming data collectors provided in this embodiment of the invention. This automatic resource optimization method for streaming data collectors is applied to the device shown in Figure 1. This device includes a sampling executor, a flow monitor, a latency monitor, an algorithm executor, and a table allocator. By periodically monitoring and analyzing key parameters such as input rate, data processing rate, and total backlog, it calculates the optimal number of tables to optimize collection efficiency. Then, based on the comparison results, it dynamically adjusts the number of tables currently being processed: when increased processing capacity is needed, a corresponding number of new tables are added from the upstream database; otherwise, redundant tables are removed, and the collection executor is restarted to ensure the configuration changes take effect. This method not only effectively avoids system bottlenecks caused by data processing delays but also prevents resource waste caused by over-configuration, achieving the best balance between resource utilization efficiency and system stability. This improves data processing speed and accuracy, maximizes the utilization of collector resources, and ultimately enhances the overall system performance.

[0043] Figure 2 is a flowchart illustrating the automatic resource optimization method for streaming data acquisition provided in an embodiment of the present invention. As shown in Figure 2, the method includes the following steps S110 to S140.

[0044] S110: Receive the list of table names and calculate the number of tables currently in use.

[0045] In this embodiment, the current table count refers to the number of database tables that the acquisition executor is currently processing and writing its binlog data to in the downstream system.

[0046] Specifically, the data collection operation is started periodically to read binlog data from the upstream database and write it to the downstream system to obtain data processing rate, input rate and total backlog of records in order to obtain monitoring data.

[0047] The data collection executor first receives a list of table names from the table allocator. This list contains the names of all tables that the data collection executor is responsible for processing.

[0048] Based on the received list of table names, the data acquisition executor connects to the upstream database and begins reading the corresponding binlog (binary log) data for these tables. The binlog records all operations that modify the database, which is extremely useful for real-time data synchronization and recovery.

[0049] The data acquisition executor processes the binlog data it reads and then writes it to the downstream system. This "downstream system" can be a big data analytics platform, a data warehouse, or other types of storage system.

[0050] Through the process described above, the data acquisition executor is actually processing data from multiple tables. The "current number of tables" mentioned here refers to the number of tables for which the data acquisition executor is processing binlog data and writing it to the downstream system at any given point in time. This represents the current workload of the data acquisition executor.

[0051] Furthermore, to further optimize resource utilization efficiency and ensure system stability, the data acquisition executor periodically acquires its data processing rate (Ps), input rate (Ins), and total backlog (B). This monitoring data provides the algorithm with necessary information to calculate the optimal number of tables (Tn'), thereby dynamically adjusting the number of tables allocated to the data acquisition executor and ensuring an optimal balance between performance and resources under a fixed resource configuration. This mechanism effectively addresses changes in traffic and data size over different time periods, avoiding data latency or resource waste caused by improper configuration.

[0052] S120. Start the data acquisition operation to read binlog data from the upstream database and write it to the downstream system to obtain the data processing rate, input rate and total backlog of records in order to obtain monitoring data.

[0053] Specifically, the data collection operation is started periodically to read binlog data from the upstream database and write it to the downstream system to obtain data processing rate, input rate and total backlog of records in order to obtain monitoring data.

[0054] In this embodiment, the monitoring data refers to the data processing rate (Ps), input rate (Ins), and total backlog (B) acquired from the acquisition actuator at regular intervals (e.g., per second). This data is used to evaluate the real-time operating status and efficiency of the streaming acquisition unit. Specifically, the process of periodically acquiring the data processing rate, input rate, and total backlog is as follows:

[0055] Data processing rate (Ps): This refers to the amount of binlog data that the data acquisition actuator can process and successfully write to the downstream system per unit of time. This metric directly reflects the current processing capacity of the data acquisition actuator.

[0056] Input rate (Ins): This refers to the amount of binlog data generated by the upstream database system per unit time that needs to be processed by the acquisition executor. This metric helps to understand the speed at which data flows into the acquisition executor.

[0057] Total backlog entries (B): This represents the total amount of binlog data that has not yet been processed by the data acquisition executor. An increase in the backlog entries may indicate that the data acquisition executor is facing a bottleneck or insufficient resources.

[0058] By monitoring the three key performance indicators mentioned above, the algorithm can analyze the monitoring data and, combined with the existing number of tables (Tn), calculate the optimal number of tables to allocate (Tn'). This not only ensures that data can be transmitted from upstream to downstream in a timely and efficient manner, but also maximizes the resource utilization of the acquisition executor, achieving a balance between resource consumption and performance. This dynamic adjustment mechanism is crucial for maintaining the stability and efficiency of the streaming acquisition process.

[0059] S130. Calculate the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result.

[0060] In this embodiment, the calculation result refers to the optimal number of tables Tn' calculated using a specific algorithm based on monitoring data (including input rate Ins, total backlog B, and data processing rate Ps) and the currently allocated number of tables Tn, in order to optimize the resource utilization efficiency and stability of the streaming data collector. Specifically, this process is performed at set intervals (e.g., every 10 minutes) based on the monitoring data and the current number of tables to determine the optimal number of tables Tn'.

[0061] Specifically, based on the monitoring data and the current number of tables, the optimal number of tables is calculated according to the set interval time to optimize the collection efficiency and obtain the calculation result.

[0062] use The optimal number of tables is calculated to optimize data collection efficiency, resulting in the calculation results, where T... n ′The optimal number of tables; Ins is the input rate; B is the total backlog; Er is the settling time; P s T represents the data processing rate. n This represents the current number of items in the table.

[0063] The purpose of this formula is to dynamically adjust the number of tables allocated to a data collector by considering its current workload (reflected by the input rate and the number of backlogged records) and its processing capacity (reflected by the data processing rate). This ensures that, with a fixed resource configuration, the collector's processing capacity is fully utilized, resource idleness is reduced, and data latency issues caused by overload are avoided, thus achieving a balance between resource utilization and performance.

[0064] Therefore, this dynamic adjustment mechanism allows the streaming data collector to automatically adjust to the most suitable number of tables under different workload conditions, achieving an efficient and stable data acquisition process. This not only improves the overall efficiency of data processing but also enhances the system's adaptability to different data flows.

[0065] S140. Readjust the data collection table list based on the calculation results and restart the data collection job.

[0066] Specifically, the number of current tables is increased or decreased based on the calculation results.

[0067] In step S140, the system adjusts the number of tables handled by the current streaming collector based on the calculation results provided by the algorithm. This process aims to optimize resource utilization and ensure the timeliness of data processing, while also ensuring system stability.

[0068] In one embodiment, step S140 described above may include steps S141 to S142.

[0069] S141. Compare the calculation result with the current table quantity to obtain a comparison result.

[0070] In this embodiment, the comparison result refers to the difference between the calculated ideal number of tables (Tn') and the currently actually allocated number of tables (Tn), which is used to determine whether and how to adjust the task configuration of the data collector. Specifically, if Tn' is greater than Tn, it indicates that the number of tables needs to be increased to make full use of resources; if Tn' is less than Tn, the number of tables needs to be reduced to avoid excessive resource consumption and ensure system stability and efficiency.

[0071] First, the system needs to compare the calculation result (Tn') given by the algorithm with the number of tables (Tn) currently allocated to the acquisition executor. The main purpose of this step is to determine whether the current configuration needs to be adjusted to achieve better performance and efficiency. Specifically:

[0072] Calculation result (Tn'): This is obtained by the algorithm based on monitoring data (such as input rate Ins, backlog data volume B, processing rate Ps, etc.) through a series of complex operations, representing the number of tables that the current acquisition executor should be responsible for in order to achieve the best resource utilization efficiency.

[0073] Current number of tables (Tn): Refers to the number of tables that have been currently allocated to this acquisition executor.

[0074] The result of the comparison can be one of the following three situations:

[0075] When Tn' > Tn, it means that the current acquisition executor can handle more tasks, that is, the number of tables should be increased.

[0076] When Tn' < Tn, it indicates that the current acquisition executor is overloaded or there is resource waste, and the number of tables should be reduced.

[0077] When Tn' = Tn, it shows that the current configuration is already in the optimal state and no adjustment is required.

[0078] S142. Increase or decrease the current number of tables according to the comparison result, and restart the acquisition executor to make the change take effect.

[0079] Specifically, when the difference between the two in the comparison result is greater than zero, add the corresponding number of new tables from the upstream database to the current number of tables; if the difference is less than zero, remove the corresponding number of tables from the current number of tables, and restart the acquisition executor to ensure that the adjusted configuration takes effect.

[0080] Once the comparison result is determined, the next step is the actual adjustment operation. This stage includes the following key actions:

[0081] Increase the number of tables: If the calculation result shows that the number of tables needs to be increased (i.e., Tn' - Tn > 0), select new tables that are not occupied by other acquisition executors from the upstream database and add them to the task list of the current acquisition executor. This can maximize its workload without exceeding the resource limit of the acquisition executor.

[0082] Reduce the number of tables: On the contrary, if it is found that the number of tables responsible for by the current acquisition executor is too large (i.e., Tn' - Tn < 0), then remove the corresponding number of tables from the current task list. These tables may be reallocated to other acquisition executors with more abundant resources to balance the load of the entire system.

[0083] Restarting the data collection executor: Whether a new table has been added or the number of existing tables has been reduced, the data collection executor needs to be restarted for the changes to take effect. This is because the new configuration can only be correctly loaded and applied after the data collection executor has been restarted.

[0084] In this way, the present invention achieves dynamic management and optimization of streaming data acquisition resources, which not only improves the speed and efficiency of data processing but also avoids resource waste and ensures system stability and reliability. This method is particularly suitable for application scenarios with large fluctuations in data traffic and a large number of data tables.

[0085] The aforementioned automatic resource optimization method for streaming data collectors ensures high efficiency and accuracy in data processing by dynamically adjusting the number of tables read from the upstream database and written to the downstream system. First, it receives a list of current table names to determine the existing task configuration and acquires monitoring data, including data processing rate, input rate, and total backlog, to assess system performance and load. Then, based on this information, it calculates the optimal number of tables to effectively handle data backlog, avoid delays, and prevent waste caused by over-allocation of resources. Finally, it automatically adjusts the number of tables based on the calculation results, achieving the best balance between resource utilization efficiency and system stability. This not only improves the speed and accuracy of single data processing but also maximizes the utilization of the collector's resources, ultimately improving the overall system's operational efficiency and reliability.

[0086] Figure 3 is a schematic block diagram of an automatic resource optimization device 300 for a streaming data collector according to an embodiment of the present invention. As shown in Figure 3, corresponding to the above-described automatic resource optimization method for a streaming data collector, the present invention also provides an automatic resource optimization device 300 for a streaming data collector. This automatic resource optimization device 300 includes units for executing the above-described automatic resource optimization method for a streaming data collector, and the device can be configured in a server. Specifically, referring to Figure 3, the automatic resource optimization device 300 for a streaming data collector includes a data acquisition execution unit 301, a monitoring unit 302, a calculation unit 303, and a table allocation unit 304.

[0087] The data acquisition execution unit 301 is used to receive a list of table names and calculate the current number of tables; the monitoring unit 302 is used to start a data acquisition job to read binlog data from the upstream database and write it to the downstream system, and obtain the data processing rate, input rate and total backlog of records to obtain monitoring data; the calculation unit 303 is used to calculate the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result; the table allocation unit 304 is used to readjust the data acquisition table list based on the calculation result and restart the data acquisition job.

[0088] In one embodiment, the monitoring unit 302 is used to periodically start a data acquisition operation to read binlog data from the upstream database and write it to the downstream system, and obtain data processing rate, input rate and total backlog number to obtain monitoring data.

[0089] In one embodiment, the calculation unit 303 is used to calculate the optimal number of tables based on the monitoring data and the current number of tables according to a set interval time to optimize the collection efficiency and obtain the calculation result.

[0090] In one embodiment, the computing unit 303 is used to employ The optimal number of tables is calculated to optimize data collection efficiency, resulting in the calculation results, where T... n ′ The optimal number of tables; Ins is the input rate; B is the total backlog; Er is the settling time; P s T represents the data processing rate. n This represents the current number of items in the table.

[0091] In one embodiment, the table allocation unit 304 is used to increase or decrease the current number of tables based on the calculation result.

[0092] In one embodiment, the table allocation unit 304 includes:

[0093] The comparison subunit is used to compare the calculation result with the current table quantity to obtain a comparison result; the tuning subunit is used to increase or decrease the current table quantity according to the comparison result and restart the data acquisition executor so that the changes take effect.

[0094] In one embodiment, the tuning subunit is configured to add a corresponding number of new tables from the upstream database to the current table count when the difference between the two is greater than zero; and to remove a corresponding number of tables from the current table count and restart the data collection executor if the difference between the two is less than zero, so as to ensure that the adjusted configuration takes effect.

[0095] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned automatic resource optimization device 300 for streaming data acquisition and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0096] The above-mentioned automatic resource optimization device 300 for streaming data acquisition can be implemented as a computer program, which can run on the computer device shown in Figure 4.

[0097] Please refer to Figure 4, which is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0098] Referring to Figure 4, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0099] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform an automatic resource optimization method for a streaming data acquisition unit.

[0100] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0101] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an automatic optimization method for streaming data acquisition resources.

[0102] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the structure shown in Figure 4 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0103] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0104] Receive a list of table names and calculate the current number of tables; start a data collection job to read binlog data from the upstream database and write it to the downstream system, obtain the data processing rate, input rate, and total backlog of records to obtain monitoring data; calculate the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result; readjust the list of collection tables based on the calculation result and restart the data collection job.

[0105] In one embodiment, when the processor 502 implements the steps of starting the data acquisition job to read binlog data from the upstream database and write it to the downstream system, and obtaining the data processing rate, input rate, and total backlog of records to obtain monitoring data, the specific implementation is as follows:

[0106] The data collection job is started periodically to read binlog data from the upstream database and write it to the downstream system. The data processing rate, input rate and total backlog number are obtained to obtain monitoring data.

[0107] In one embodiment, when the processor 502 calculates the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result, the processor 502 specifically implements the following steps:

[0108] Based on the monitoring data and the current number of tables, the optimal number of tables is calculated according to the set interval to optimize the collection efficiency and obtain the calculation result.

[0109] In one embodiment, when the processor 502 calculates the optimal number of tables based on the monitoring data and the current number of tables according to a set interval to optimize data collection efficiency and obtain the calculation result, the processor specifically implements the following steps:

[0110] use The optimal number of tables is calculated to optimize data collection efficiency, resulting in the calculation results, where T... n ′ The optimal number of tables; Ins is the input rate; B is the total backlog; Er is the settling time; P s T represents the data processing rate. n This represents the current number of items in the table.

[0111] In one embodiment, when the processor 502 implements the step of readjusting the acquisition table list based on the calculation results and restarting the acquisition job, it specifically implements the following steps:

[0112] The number of current tables may be increased or decreased based on the calculation results.

[0113] In one embodiment, when implementing the step of increasing or decreasing the current table quantity based on the calculation result, the processor 502 specifically implements the following steps:

[0114] The calculation result is compared with the current table count to obtain a comparison result; the current table count is increased or decreased based on the comparison result, and the data collection executor is restarted to make the changes take effect.

[0115] In one embodiment, when the processor 502 implements the step of increasing or decreasing the current table quantity based on the comparison result and restarting the data acquisition executor to make the change take effect, it specifically implements the following steps:

[0116] If the difference between the two is greater than zero, add the corresponding number of new tables from the upstream database to the current table count; if the difference is less than zero, remove the corresponding number of tables from the current table count and restart the data collection executor to ensure that the adjusted configuration takes effect.

[0117] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0118] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0119] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps:

[0120] Receive a list of table names and calculate the current number of tables; start a data collection job to read binlog data from the upstream database and write it to the downstream system, obtain the data processing rate, input rate, and total backlog of records to obtain monitoring data; calculate the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result; readjust the list of collection tables based on the calculation result and restart the data collection job.

[0121] In one embodiment, when the processor executes the computer program to implement the step of starting the data acquisition job by reading binlog data from the upstream database and writing it to the downstream system, and obtaining the data processing rate, input rate, and total backlog of records to obtain monitoring data, the processor specifically implements the following steps:

[0122] The data collection job is started periodically to read binlog data from the upstream database and write it to the downstream system. The data processing rate, input rate and total backlog number are obtained to obtain monitoring data.

[0123] In one embodiment, when the processor executes the computer program to calculate the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result, the processor specifically implements the following steps:

[0124] Based on the monitoring data and the current number of tables, the optimal number of tables is calculated according to the set interval to optimize the collection efficiency and obtain the calculation result.

[0125] In one embodiment, when the processor executes the computer program to calculate the optimal number of tables based on the monitoring data and the current number of tables at set intervals to optimize data collection efficiency and obtain the calculation result, the specific steps are as follows:

[0126] use The optimal number of tables is calculated to optimize data collection efficiency, resulting in the calculation results, where T... n ′ The optimal number of tables; Ins is the input rate; B is the total backlog; Er is the settling time; P s T represents the data processing rate. n This represents the current number of items in the table.

[0127] In one embodiment, when the processor executes the computer program to implement the steps of readjusting the acquisition table list based on the calculation results and restarting the acquisition job, it specifically implements the following steps:

[0128] The number of current tables may be increased or decreased based on the calculation results.

[0129] In one embodiment, when the processor executes the computer program to implement the step of increasing or decreasing the current table quantity based on the calculation result, it specifically implements the following steps:

[0130] The calculation result is compared with the current table count to obtain a comparison result; the current table count is increased or decreased based on the comparison result, and the data collection executor is restarted to make the changes take effect.

[0131] In one embodiment, when the processor executes the computer program to increase or decrease the current table quantity based on the comparison result and restarts the data acquisition executor so that the change takes effect, the processor specifically implements the following steps:

[0132] If the difference between the two is greater than zero, add the corresponding number of new tables from the upstream database to the current table count; if the difference is less than zero, remove the corresponding number of tables from the current table count and restart the data collection executor to ensure that the adjusted configuration takes effect.

[0133] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0135] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0136] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0138] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for automatic resource optimization in streaming data acquisition, characterized in that, include: Receive a list of table names to obtain the current number of tables; The data collection operation is initiated to read binlog data from the upstream database and write it to the downstream system. The data processing rate, input rate, and total backlog of records are obtained to acquire monitoring data. Based on the monitoring data and the current number of tables, the optimal number of tables is calculated to obtain the calculation result. The data collection table list is readjusted based on the calculation results, and the data collection job is restarted.

2. The automatic resource optimization method for streaming data acquisition machines according to claim 1, characterized in that, The process of initiating a data collection operation to read binlog data from the upstream database and write it to the downstream system, and obtaining data processing rate, input rate, and total backlog of records to obtain monitoring data, includes: initiating a data collection operation to read binlog data from the upstream database and write it to the downstream system, and periodically initiating a data collection operation to read binlog data from the upstream database and write it to the downstream system, obtaining data processing rate, input rate, and total backlog of records to obtain monitoring data.

3. The automatic resource optimization method for streaming data acquisition machines according to claim 1, characterized in that, The step of calculating the optimal number of tables based on the monitoring data and the current number of tables to obtain the calculation result includes: calculating the optimal number of tables based on the monitoring data and the current number of tables according to a set interval to optimize the collection efficiency and obtain the calculation result.

4. The automatic resource optimization method for streaming data acquisition machines according to claim 3, characterized in that, The step of calculating the optimal number of tables based on the monitoring data and the current number of tables according to a set interval to optimize data collection efficiency and obtain the calculation result includes: using... The optimal number of tables is calculated to optimize data collection efficiency, resulting in the calculation results, where T... n ′ The optimal number of tables; Ins is the input rate; B is the total backlog; Er is the settling time; P s T represents the data processing rate. n This represents the current number of items in the table.

5. The automatic resource optimization method for streaming data acquisition machines according to claim 1, characterized in that, The step of readjusting the data collection list based on the calculation results and restarting the data collection job includes: regenerating the data collection list based on the calculation results and restarting the data collection job.

6. The automatic resource optimization method for streaming data acquisition machines according to claim 5, characterized in that, The step of regenerating the collection list based on the calculation results and restarting the collection job includes: comparing the calculation results with the current table count to obtain a comparison result; increasing or decreasing the current table count based on the comparison result, and restarting the collection executor to make the changes take effect.

7. The automatic resource optimization method for streaming data acquisition machines according to claim 6, characterized in that, The step of increasing or decreasing the current number of tables based on the comparison results and restarting the data collection executor to make the changes take effect includes: when the difference between the two is greater than zero, adding a corresponding number of new tables from the upstream database to the current number of tables; if the difference is less than zero, removing a corresponding number of tables from the current number of tables and restarting the data collection executor to ensure that the adjusted configuration takes effect.

8. An automatic resource optimization device for streaming data acquisition, characterized in that, include: The data collection and execution unit is used to receive a list of table names and calculate the number of tables currently in use. The monitoring unit is used to initiate a data acquisition operation to read binlog data from the upstream database and write it to the downstream system, and to obtain data processing rate, input rate and total backlog number to obtain monitoring data. The calculation unit is used to calculate the optimal number of tables based on the monitoring data and the current number of tables, so as to obtain the calculation result; The table allocation unit is used to readjust the collection table list based on the calculation results and restart the collection job.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images