Data extraction adaptive optimization method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a data extraction adaptive optimization method and device, computer equipment and a storage medium. The method comprises the following steps: scanning a monitoring index of a database server; determining each factor of the thread according to the monitoring index; performing data extraction according to each factor of the thread; and when the data extraction is not completed, dynamically collecting the monitoring indexes of the monitoring database server, adjusting the factors of the thread according to the dynamically collected monitoring indexes, and executing the data extraction according to the factors of the thread. By implementing the method provided by the embodiment of the invention, the benefit maximization of data extraction can be achieved as far as possible by intelligently adjusting the extraction strategy under the condition of not influencing a service system, and the stability and efficiency of the whole system are improved.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and more specifically to a data extraction adaptive optimization method, device, computer equipment and storage medium. Background Art

[0002] Data extraction technology refers to the process of extracting, cleaning, and transferring data from one or more data sources. It is a key component of data management tasks such as data integration and data warehousing. Data extraction technology aims to effectively capture data from the source system and organize and transfer it to the target end for various scenarios.

[0003] Traditional data extraction technology usually uses real-time or periodic methods to transmit data. Real-time extraction maintains the extraction between data sources by transmitting data instantly. Under high load, real-time extraction may cause server performance to reach a bottleneck, and in severe cases, it may directly cause the business system to crash. The periodic extraction method transmits data at a predetermined time interval. On the one hand, it may cause data transmission to be untimely and have a high delay. On the other hand, if the interval is long or the amount of data generated by the data source is large, the periodic extraction method may not be able to process and transmit all the data in time, causing data to accumulate in the system.

[0004] Therefore, it is necessary to design a new method to maximize the benefits of data extraction and improve the stability and efficiency of the overall system by intelligently adjusting the extraction strategy without affecting the business system. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a data extraction adaptive optimization method, device, computer equipment and storage medium.

[0006] To achieve the above object, the present invention adopts the following technical solution: a data extraction adaptive optimization method, comprising:

[0007] Scan the monitoring indicators of the database server;

[0008] Determine various factors of the thread according to the monitoring indicators;

[0009] Extract data based on various factors of the thread;

[0010] When data extraction is not completed, monitoring indicators of the monitoring database server are dynamically collected, various factors of the thread are adjusted according to the dynamically collected monitoring indicators, and data extraction is performed according to the various factors of the thread.

[0011] Its further technical solution is: the monitoring indicators include disk IO, number of database connections, network utilization, database throughput, memory utilization, and CPU utilization.

[0012] A further technical solution is: determining various factors of the thread according to the monitoring index includes:

[0013] Performing weighted summation on the monitoring indicators to obtain a comprehensive utilization rate;

[0014] Various factors of the thread are calculated according to the comprehensive utilization rate.

[0015] Its further technical solution is: the various factors of the thread include the number of threads, the number of coroutine subtasks and the data extraction batch.

[0016] A further technical solution is: the number of threads is calculated as follows: number of threads = (1-comprehensive utilization rate × thread coefficient;

[0017] The number of coroutine subtasks is calculated as follows: number of coroutine subtasks = (1-comprehensive utilization rate) × coroutine coefficient;

[0018] The calculation method of the data extraction batch is batch size = (1-comprehensive utilization rate) × batch coefficient.

[0019] A further technical solution is: when the data extraction is not completed, the monitoring indicators of the monitoring database server are dynamically collected, the various factors of the thread are adjusted according to the dynamically collected monitoring indicators, and the data extraction is performed according to the various factors of the thread, including:

[0020] When data extraction is not completed, dynamically collect and monitor the monitoring indicators of the database server, and calculate the comprehensive utilization rate after dynamic collection;

[0021] Determine the fluctuation degree of the comprehensive utilization rate according to the comprehensive utilization rate and the comprehensive utilization rate after dynamic collection;

[0022] Calculate various factors of the thread according to the comprehensive utilization rate after dynamic collection to obtain the secondary calculation result;

[0023] Determine the adjustment strategy based on the fluctuation level and the secondary calculation result;

[0024] The various factors of the thread are adjusted according to the adjustment strategy, and data extraction is performed according to the various factors of the thread.

[0025] Its further technical solution is: the adjustment strategy includes reducing the collection frequency when the comprehensive utilization rate after dynamic collection increases relative to the comprehensive utilization rate; increasing the collection frequency when the comprehensive utilization rate after dynamic collection decreases relative to the comprehensive utilization rate; and when the fluctuation degree is not greater than the set threshold, updating all factors of the thread; when the fluctuation degree is greater than the set threshold, not updating the number of threads, but updating other factors of the thread.

[0026] The present invention also provides a data extraction adaptive optimization device, comprising:

[0027] The first scanning unit is used to scan the monitoring indicators of the database server;

[0028] A factor determination unit, used to determine various factors of the thread according to the monitoring indicator;

[0029] An extraction unit, used to extract data according to various factors of the thread;

[0030] The dynamic adjustment unit is used to dynamically collect monitoring indicators of the monitoring database server when data extraction is not completed, adjust various factors of the thread according to the dynamically collected monitoring indicators, and execute data extraction according to the various factors of the thread.

[0031] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0032] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0033] The beneficial effect of the present invention compared with the prior art is: the present invention determines the comprehensive utilization rate according to the monitoring indicators, and then determines the various factors of the thread according to the comprehensive utilization rate, and uses them as parameters for data extraction. During the extraction process, these monitoring indicators are dynamically collected, and various factors are adjusted in real time. Without affecting the business system, the extraction strategy is intelligently adjusted to maximize the benefits of data extraction and improve the stability and efficiency of the overall system.

[0034] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0036] Figure 1 A schematic diagram of an application scenario of the data extraction adaptive optimization method provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a flow chart of a data extraction adaptive optimization method provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a sub-process of a data extraction adaptive optimization method provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a sub-process of a data extraction adaptive optimization method provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a real-time effect diagram provided by an embodiment of the present invention;

[0041] Figure 6 A schematic block diagram of a data extraction adaptive optimization device provided by an embodiment of the present invention;

[0042] Figure 7 A schematic block diagram of a factor determination unit of a data extraction adaptive optimization device provided by an embodiment of the present invention;

[0043] Figure 8 A schematic block diagram of a dynamic adjustment unit of a data extraction adaptive optimization device provided by an embodiment of the present invention;

[0044] Fig. 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0047] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0048] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the data extraction adaptive optimization method provided in an embodiment of the present invention. Figure 2 A schematic flow chart of a data extraction adaptive optimization method provided in an embodiment of the present invention. The data extraction adaptive optimization method is applied to a database server. The database server is integrated with a MySql business library, a data extraction tool, and a MySql desensitization library. Taking the database MySQL as an example, when extracting data from the MySQL business library to the desensitization library in the traditional way, a fixed number of threads is usually used for extraction. However, there are some problems with fixed thread extraction. If the number of threads designed is too large, it will not only directly affect the normal operation of the business system, but also cause IO intensiveness, and the speed of transmission and extraction will decrease instead; if the number of threads designed is too small, it is highly likely that the growth rate of business data will be much higher than the speed of data extraction, which will not only cause a large amount of data to accumulate, but also greatly increase the complexity of data extraction, and may also waste the performance of the machine. The method of this embodiment provides the ability to dynamically perceive the load of the MySQL library, and dynamically adjust the technical parameters for extracting data according to the real-time load conditions. By dynamically monitoring the hardware resources and load conditions of the server, the buffer size, query cache, and concurrent connection number parameters are adjusted in real time to maximize the use of available resources and improve extraction performance. Figure 1 As shown in the figure, through dynamic tuning technology, the tool can perform intelligent optimization based on the actual load of the MySQL database, ensuring that the extraction process does not over-occupy system resources while maintaining efficient data extraction speed. This dynamic thread adjustment method can help balance the load of the MySQL business database and the performance of data extraction, and improve the stability and efficiency of the overall system.

[0050] Figure 2 FIG. 1 is a flow chart of a data extraction adaptive optimization method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S140.

[0051] S110. Scan monitoring indicators of the database server.

[0052] In this embodiment, the monitoring indicators include disk IO, number of database connections, network utilization, database throughput, memory utilization, and CPU utilization.

[0053] Among them, monitoring the disk IO situation, including read and write speed, response time, etc., can help determine the degree of disk load. Tracking the number of database connections can reflect the database load; network utilization specifically monitors network traffic, including the amount of data received and sent, which can be used to evaluate the system's network load; database throughput refers to the amount of data or the number of transactions that a database system can process per unit time. It is one of the important indicators for measuring database performance and efficiency. Tracking the system's memory utilization can reflect the system's memory usage. Monitoring the system's CPU utilization can be used as a performance indicator to evaluate the system's load.

[0054] Data extraction is an IO-intensive task, so disk IO accounts for the largest weight in the collection indicators. Coroutines are more suitable for IO-intensive tasks. Coroutines are lighter, more concurrent, and have lower overhead than traditional threads, which can further improve the efficiency of data extraction.

[0055] Install and initialize the data extraction tool on the database server to automatically collect the monitoring indicators of the database server and further calculate the comprehensive utilization rate.

[0056] S120: Determine various factors of the thread according to the monitoring indicator.

[0057] In one embodiment, see Figure 3 , the above-mentioned step S120 may include steps S121 to S122.

[0058] S121. Perform weighted summation on the monitoring indicators to obtain a comprehensive utilization rate.

[0059] In this embodiment, the comprehensive utilization rate refers to the comprehensive usage of the database server.

[0060] Evaluate the comprehensive utilization of the database server and calculate a reasonable number of threads. The number of threads is also related to the hardware configuration of the database server. Generally, when the number of threads enabled is consistent with the number of CPU cores, the idle resources of the database server can be fully utilized. The number of threads enabled is subject to the following restrictions:

[0061] Because when there are too many threads, the CPU cache hit rate will drop significantly, and multiple threads will often compete for a CPU cache line;

[0062] The read and write operations of the opened thread are different. Secondly, the throughput of the read operation can increase linearly, but the write operation cannot because there is write competition in the write operation.

[0063] Large amounts of read and write memory bandwidth are also potential bottlenecks.

[0064] The overhead of multithreading is often cache failure, locks, and data contention. These problems limit the ability of modern software systems to grow linearly with the increase of CPU cores. Throughput does not grow linearly, especially starting with 16 cores, and even starts to decline. Therefore, even for a 48-core machine, the maximum number of threads enabled by the present invention is limited to 16. In other cases, the corresponding maximum number of threads is the number of CPU cores to ensure full utilization of idle resources of the database server. The specific number of threads enabled is determined based on the above indicators. The indicators are weighted averaged, and the weights are shown in Table 1 to calculate the comprehensive utilization rate. The number of threads enabled is finally determined (the minimum number of threads is 1).

[0065] Table 1. Weight values ​​of monitoring indicators

[0066] index Weight Disk IO 3 Number of database connections 2 Network utilization 1 Database throughput 2 Memory Utilization 1 CPU Utilization 1

[0067] S122. Calculate various factors of the thread according to the comprehensive utilization rate.

[0068] In this embodiment, the various factors of the thread include the number of threads, the number of coroutine subtasks, and the data extraction batch.

[0069] In addition to the above content, it also includes the collection frequency. The default collection frequency in this embodiment is once every 2 minutes, and the default collection is once at startup.

[0070] In this embodiment, the calculation method of the number of threads, i.e., the linear relationship, is the number of threads = (1-comprehensive utilization rate) × thread coefficient;

[0071] The thread number threshold segment relationship is as follows:

[0072] The number of CPU cores is obtained by calling the operating system API directly, and the minimum number of threads defaults to 1;

[0073] Comprehensive utilization rate <20%: Number of threads = maximum number of threads (number of CPU cores);

[0074] 20%<comprehensive utilization rate<50%:

[0075] 50%<comprehensive utilization rate<70%:

[0076] Comprehensive utilization rate>70%: Number of threads = minimum number of threads (1);

[0077] Comprehensive utilization rate>80%: suspend the extraction task.

[0078] In this embodiment, the linear relationship between the number of threads, the relationship between the number of threads and the comprehensive utilization rate, and the coroutine subtasks and batches are all similar linear relationships. The minimum data are all set default values, and the number of CPU cores is directly obtained from the operating system API interface.

[0079] The calculation method of the number of coroutine subtasks, i.e., the linear relationship, is the number of coroutine subtasks = (1-comprehensive utilization rate) × coroutine coefficient;

[0080] In this embodiment, the coroutine number threshold is segmented as follows:

[0081] The default minimum number of coroutine subtasks is 1;

[0082] Comprehensive utilization rate <20%: number of coroutine subtasks = maximum number of coroutine subtasks (number of CPU cores) × 1.5;

[0083] 20%<comprehensive utilization rate<50%:

[0084] 50%<comprehensive utilization rate<70%:

[0085] Comprehensive utilization rate>70%: Number of coroutine subtasks = minimum number of coroutine subtasks (1);

[0086] Comprehensive utilization rate>80%: suspend the extraction task.

[0087] In this embodiment, default values ​​are set for the above-mentioned related parameters.

[0088] The calculation method of the data extraction batch, that is, the linear relationship, is batch size = (1-comprehensive utilization rate) × batch coefficient.

[0089] In this embodiment,

[0090] Threshold segmentation:

[0091] The default minimum batch size is 1;

[0092] Comprehensive utilization rate < 20%: batch size = maximum batch size (50,000 rows);

[0093] 20%<comprehensive utilization rate<50%: batch size = (1-comprehensive utilization rate) ×

[0094] Maximum batch size;

[0095] 50%<comprehensive utilization rate<70%: batch size=

[0096] (1-comprehensive utilization rate) × maximum batch size - 5000 rows;

[0097] Comprehensive utilization rate>70%: batch size = minimum batch size (1000 rows);

[0098] Comprehensive utilization rate>80%: suspend the extraction task.

[0099] In this embodiment, default values ​​are set for the above-mentioned related parameters.

[0100] S130, extracting data according to various factors of the thread.

[0101] In this embodiment, the extraction tool is initialized according to the calculated factors to start extracting data.

[0102] Specifically, corresponding tools are used to extract data according to the number of threads, the number of coroutine subtasks, the data extraction batches, and the collection frequency.

[0103] Taking a 4-core CPU database server with an initial comprehensive utilization rate of less than 20% as an example, we can get the initial number of extraction threads to be 4, the number of coroutine subtasks to be 6, and the batch size to be the maximum batch for data extraction.

[0104] S140. When data extraction is not completed, dynamically collect monitoring indicators of the monitoring database server, adjust various factors of the thread according to the dynamically collected monitoring indicators, and perform data extraction according to the various factors of the thread.

[0105] When data extraction is completed, the above process ends.

[0106] In one embodiment, see Figure 4 , the above-mentioned step S140 may include steps S141 to S145.

[0107] S141. When data extraction is not completed, dynamically collect monitoring indicators of the monitoring database server and calculate the comprehensive utilization rate after dynamic collection.

[0108] In this embodiment, the comprehensive utilization rate can be calculated by referring to step S121, which will not be described in detail here.

[0109] S142. Determine the fluctuation degree of the comprehensive utilization rate according to the comprehensive utilization rate and the comprehensive utilization rate after dynamic collection.

[0110] The degree of fluctuation can be determined by dividing the difference between the comprehensive utilization rate after dynamic collection and the comprehensive utilization rate by the comprehensive utilization rate to obtain a percentage.

[0111] In this embodiment, a short fluctuation of the percentage within the range of 0 to 5% indicates a small fluctuation of the comprehensive utilization rate. A percentage greater than 5% indicates a large fluctuation of the comprehensive utilization rate.

[0112] S143. Calculate various factors of the thread according to the comprehensive utilization rate after dynamic collection to obtain a secondary calculation result.

[0113] In this embodiment, the secondary settlement result refers to various factors of the thread calculated according to the comprehensive utilization rate after dynamic collection. Please refer to the calculation method of step S122, which will not be repeated here.

[0114] S144. Determine an adjustment strategy according to the fluctuation degree and the secondary calculation result.

[0115] In this embodiment, the collection frequency is not included in the factor, and the adjustment strategy includes reducing the collection frequency when the comprehensive utilization rate after dynamic collection increases relative to the comprehensive utilization rate; increasing the collection frequency when the comprehensive utilization rate after dynamic collection decreases relative to the comprehensive utilization rate; and when the fluctuation degree is not greater than the set threshold, all factors of the thread are updated; when the fluctuation degree is greater than the set threshold, the number of threads is not updated, and other factors of the thread are updated.

[0116] In this embodiment, when the comprehensive utilization rate after dynamic collection is less than 20%, it is a low load range of the database server, 20% to 50% is a normal working range, 50% to 70% is a higher load range, 70% to 80% is a high load range, and above 80% is an extremely high load range.

[0117] S145. Adjust various factors of the thread according to the adjustment strategy, and perform data extraction according to the various factors of the thread.

[0118] When dynamically collecting monitoring indicators, the actual performance of the database server should be affected as little as possible, and the collection frequency should be adjusted accordingly. When the database server load is high, the collection frequency will dynamically decrease, and when the database server load is average or low, the collection frequency will dynamically increase. Dynamically adjusting the frequency reduces the impact on the database server.

[0119] The frequency of dynamically adjusting the extraction performance should not be too frequent. Frequent adjustment may increase the overhead of context switching and lead to performance degradation. Therefore, when dynamically adjusting the extraction performance, the overall utilization rate fluctuates slightly, but the calculated number of threads changes and is not adjusted.

[0120] In this embodiment, the collection frequency is not included in the factor, and the standard for reduction and increase is whether the fluctuation of the comprehensive utilization rate is greater than 5%. If it is greater, the collection frequency is adjusted to be reduced, and vice versa.

[0121] The real-time effect diagram of extracting data within a period of time on the MySQL database server using the method of this embodiment is as follows: Figure 5 shown.

[0122] Specifically, the frequency of collection is actually linked to the fluctuation of the comprehensive utilization rate. The tool considers that the fluctuation of the comprehensive utilization rate is greater than 5%, which is considered to be a large fluctuation. For example, the comprehensive utilization rate of the first collection is 57%, and the second is 50%. The interval between the first and second times is 2 minutes. The fluctuation here exceeds 5%, and the collection interval is shortened by half. The next collection is 1 minute interval, and the minimum interval is 30s. On the contrary, if its fluctuation is small, it will be gradually restored, and the maximum interval is 2 minutes. The adjustment intervals of other factors are determined by the comprehensive utilization rate. The document has the adjustment intervals of the factors corresponding to the changes in the comprehensive utilization rate less than 20%, 20% < comprehensive utilization rate < 50%, 70% < comprehensive utilization rate < 80%, and comprehensive utilization rate > 80%; while traditional data extraction is to open fixed threads by default (manually set), there is no coroutine subtask, and the collection batch is set manually. The innovation of this tool lies in the self-adjustment of threads, coroutine subtasks, extraction batches, and extraction rate, so as to obtain the expected optimal extraction performance without affecting the business.

[0123] In addition, you can also use machine learning algorithms to adjust the optimal maximum and minimum default values. Currently, the default values ​​of factors such as the number of threads, the number of coroutines, and the maximum and minimum batches of the extraction tool are fixed. You can use the minimum default value when the tool just starts running, without setting the maximum default value. Save the adjustment data and extraction rate of the tool operation at runtime, establish a machine learning model, use time nodes as features, and the adjustment data and extraction rate of the tool operation as labels, and train the optimal maximum and minimum default values. In this way, different devices can adapt to obtain the maximum and minimum values ​​that best suit them.

[0124] The above-mentioned data extraction adaptive optimization method determines the comprehensive utilization rate according to the monitoring indicators, and then determines the various factors of the thread according to the comprehensive utilization rate, and uses this as a parameter for data extraction. During the extraction process, these monitoring indicators are dynamically collected and various factors are adjusted in real time. Without affecting the business system, the extraction strategy is intelligently adjusted to maximize the benefits of data extraction and improve the stability and efficiency of the overall system.

[0125] Figure 6 is a schematic block diagram of a data extraction adaptive optimization device 300 provided in an embodiment of the present invention. Figure 6 As shown, corresponding to the above data extraction adaptive optimization method, the present invention also provides a data extraction adaptive optimization device 300. The data extraction adaptive optimization device 300 includes a unit for executing the above data extraction adaptive optimization method, and the device can be configured in a server. Figure 6The data extraction adaptive optimization device 300 includes a first scanning unit 301, a factor determination unit 302, an extraction unit 303 and a dynamic adjustment unit 304.

[0126] The first scanning unit 301 is used to scan the monitoring indicators of the database server; the factor determination unit 302 is used to determine the various factors of the thread according to the monitoring indicators; the extraction unit 303 is used to extract data according to the various factors of the thread; the dynamic adjustment unit 304 is used to dynamically collect and monitor the monitoring indicators of the database server when data extraction is not completed, adjust the various factors of the thread according to the dynamically collected monitoring indicators, and execute the data extraction according to the various factors of the thread.

[0127] In one embodiment, if Figure 7 As shown, the factor determination unit 302 includes a comprehensive utilization rate determination subunit 3021 and a factor calculation subunit 3022 .

[0128] The comprehensive utilization rate determination subunit 3021 is used to perform weighted summation on the monitoring indicators to obtain the comprehensive utilization rate; the factor calculation subunit 3022 is used to calculate various factors of the thread according to the comprehensive utilization rate.

[0129] In one embodiment, if Figure 8 As shown, the dynamic adjustment unit 304 includes a dynamic collection subunit 3041 , a fluctuation degree determination subunit 3042 , a calculation subunit 3043 , a strategy determination subunit 3044 and an adjustment subunit 3045 .

[0130] The dynamic collection subunit 3041 is used to dynamically collect monitoring indicators of the monitoring database server when data extraction is not completed, and calculate the comprehensive utilization rate after dynamic collection; the fluctuation degree determination subunit 3042 is used to determine the fluctuation degree of the comprehensive utilization rate based on the comprehensive utilization rate and the comprehensive utilization rate after dynamic collection; the calculation subunit 3043 is used to calculate various factors of the thread according to the comprehensive utilization rate after dynamic collection to obtain a secondary calculation result; the strategy determination subunit 3044 is used to determine the adjustment strategy according to the fluctuation degree and the secondary calculation result; the adjustment subunit 3045 is used to adjust the various factors of the thread according to the adjustment strategy, and execute the data extraction according to the various factors of the thread.

[0131] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned data extraction adaptive optimization device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0132] The data extraction adaptive optimization device 300 can be implemented in the form of a computer program. Fig. 9 Runs on the computer device shown.

[0133] See also Fig. 9 , Fig. 9 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0134] See also Fig. 9 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0135] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a data extraction adaptive optimization method.

[0136] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0137] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a data extraction adaptive optimization method.

[0138] The network interface 505 is used to communicate with other devices over the network. Fig. 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0140] Scan the monitoring indicators of the database server; determine the various factors of the thread according to the monitoring indicators; extract data according to the various factors of the thread; when the data extraction is not completed, dynamically collect and monitor the monitoring indicators of the database server, adjust the various factors of the thread according to the dynamically collected monitoring indicators, and execute the data extraction according to the various factors of the thread.

[0141] The monitoring indicators include disk IO, number of database connections, network utilization, database throughput, memory utilization, and CPU utilization.

[0142] In one embodiment, when the processor 502 implements the step of determining various factors of the thread according to the monitoring indicator, the processor 502 specifically implements the following steps:

[0143] The monitoring indicators are weighted and summed to obtain a comprehensive utilization rate; and various factors of the thread are calculated according to the comprehensive utilization rate.

[0144] Among them, the various factors of the thread include the number of threads, the number of coroutine subtasks and the data extraction batch.

[0145] The number of threads is calculated as follows: number of threads = (1-comprehensive utilization rate) × thread coefficient;

[0146] The number of coroutine subtasks is calculated as follows: number of coroutine subtasks = (1-comprehensive utilization rate) × coroutine coefficient;

[0147] The calculation method of the data extraction batch is batch size = (1-comprehensive utilization rate) × batch coefficient.

[0148] In one embodiment, when the processor 502 implements the step of dynamically collecting monitoring indicators of the monitoring database server when data extraction is not completed, adjusting various factors of the thread according to the dynamically collected monitoring indicators, and performing the step of extracting data according to various factors of the thread, the following steps are specifically implemented:

[0149] When data extraction is not completed, the monitoring indicators of the monitoring database server are dynamically collected and the comprehensive utilization rate after dynamic collection is calculated; the fluctuation degree of the comprehensive utilization rate is determined according to the comprehensive utilization rate and the comprehensive utilization rate after dynamic collection; various factors of the thread are calculated according to the comprehensive utilization rate after dynamic collection to obtain a secondary calculation result; an adjustment strategy is determined according to the fluctuation degree in combination with the secondary calculation result; various factors of the thread are adjusted according to the adjustment strategy, and data extraction is performed according to the various factors of the thread.

[0150] Among them, the adjustment strategy includes reducing the collection frequency when the comprehensive utilization rate after dynamic collection increases relative to the comprehensive utilization rate; increasing the collection frequency when the comprehensive utilization rate after dynamic collection decreases relative to the comprehensive utilization rate; and when the fluctuation degree is not greater than the set threshold, updating all factors of the thread; when the fluctuation degree is greater than the set threshold, not updating the number of threads, but updating other factors of the thread.

[0151] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0152] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0153] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0154] Scan the monitoring indicators of the database server; determine the various factors of the thread according to the monitoring indicators; extract data according to the various factors of the thread; when the data extraction is not completed, dynamically collect and monitor the monitoring indicators of the database server, adjust the various factors of the thread according to the dynamically collected monitoring indicators, and execute the data extraction according to the various factors of the thread.

[0155] The monitoring indicators include disk IO, number of database connections, network utilization, database throughput, memory utilization, and CPU utilization.

[0156] In one embodiment, when the processor executes the computer program to implement the step of determining various factors of the thread according to the monitoring indicator, the processor specifically implements the following steps:

[0157] The monitoring indicators are weighted and summed to obtain a comprehensive utilization rate; and various factors of the thread are calculated according to the comprehensive utilization rate.

[0158] Among them, the various factors of the thread include the number of threads, the number of coroutine subtasks and the data extraction batch.

[0159] The number of threads is calculated as follows: number of threads = (1-comprehensive utilization rate) × thread coefficient;

[0160] The number of coroutine subtasks is calculated as follows: number of coroutine subtasks = (1-comprehensive utilization rate) × coroutine coefficient;

[0161] The calculation method of the data extraction batch is batch size = (1-comprehensive utilization rate) × batch coefficient.

[0162] In one embodiment, when the processor executes the computer program to implement the step of dynamically collecting monitoring indicators of the monitoring database server when data extraction is not completed, adjusting various factors of the thread according to the dynamically collected monitoring indicators, and performing the step of extracting data according to various factors of the thread, the following steps are specifically implemented:

[0163] When data extraction is not completed, the monitoring indicators of the monitoring database server are dynamically collected and the comprehensive utilization rate after dynamic collection is calculated; the fluctuation degree of the comprehensive utilization rate is determined according to the comprehensive utilization rate and the comprehensive utilization rate after dynamic collection; various factors of the thread are calculated according to the comprehensive utilization rate after dynamic collection to obtain a secondary calculation result; an adjustment strategy is determined according to the fluctuation degree in combination with the secondary calculation result; various factors of the thread are adjusted according to the adjustment strategy, and data extraction is performed according to the various factors of the thread.

[0164] Among them, the adjustment strategy includes reducing the collection frequency when the comprehensive utilization rate after dynamic collection increases relative to the comprehensive utilization rate; increasing the collection frequency when the comprehensive utilization rate after dynamic collection decreases relative to the comprehensive utilization rate; and when the fluctuation degree is not greater than the set threshold, updating all factors of the thread; when the fluctuation degree is greater than the set threshold, not updating the number of threads, but updating other factors of the thread.

[0165] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0166] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0168] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0170] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. Data extraction adaptive optimization method, characterized in that: include: Scan the monitoring indicators of the database server; Determine various factors of the thread according to the monitoring indicators; Extract data based on various factors of the thread; When data extraction is not completed, monitoring indicators of the monitoring database server are dynamically collected, various factors of the thread are adjusted according to the dynamically collected monitoring indicators, and data extraction is performed according to the various factors of the thread.

2. The data extraction adaptive optimization method according to claim 1, characterized in that: The monitoring indicators include disk IO, number of database connections, network utilization, database throughput, memory utilization, and CPU utilization.

3. The data extraction adaptive optimization method according to claim 2, characterized in that: The determining of various factors of the thread according to the monitoring indicator includes: Performing weighted summation on the monitoring indicators to obtain a comprehensive utilization rate; Various factors of the thread are calculated according to the comprehensive utilization rate.

4. The data extraction adaptive optimization method according to claim 3, characterized in that: The various factors of the thread include the number of threads, the number of coroutine subtasks, and the data extraction batch.

5. The data extraction adaptive optimization method according to claim 4, characterized in that: The number of threads is calculated as follows: number of threads = (1-comprehensive utilization rate) × thread coefficient; The number of coroutine subtasks is calculated as follows: number of coroutine subtasks = (1-comprehensive utilization rate) × coroutine coefficient; The calculation method of the data extraction batch is batch size = (1-comprehensive utilization rate) × batch coefficient.

6. The data extraction adaptive optimization method according to claim 5, characterized in that: When the data extraction is not completed, dynamically collect monitoring indicators of the monitoring database server, adjust various factors of the thread according to the dynamically collected monitoring indicators, and perform data extraction according to the various factors of the thread, including: When data extraction is not completed, dynamically collect and monitor the monitoring indicators of the database server, and calculate the comprehensive utilization rate after dynamic collection; Determine the fluctuation degree of the comprehensive utilization rate according to the comprehensive utilization rate and the comprehensive utilization rate after dynamic collection; Calculate various factors of the thread according to the comprehensive utilization rate after dynamic collection to obtain the secondary calculation result; Determine the adjustment strategy based on the fluctuation level and the secondary calculation result; The various factors of the thread are adjusted according to the adjustment strategy, and data extraction is performed according to the various factors of the thread.

7. The data extraction adaptive optimization method according to claim 6, characterized in that: The adjustment strategy includes reducing the collection frequency when the comprehensive utilization rate after dynamic collection increases relative to the comprehensive utilization rate; increasing the collection frequency when the comprehensive utilization rate after dynamic collection decreases relative to the comprehensive utilization rate; and when the fluctuation degree is not greater than the set threshold, updating all factors of the thread; when the fluctuation degree is greater than the set threshold, not updating the number of threads, but updating other factors of the thread.

8. The data extraction adaptive optimization device is characterized by: include: The first scanning unit is used to scan the monitoring indicators of the database server; A factor determination unit, used to determine various factors of the thread according to the monitoring indicator; An extraction unit, used to extract data according to various factors of the thread; The dynamic adjustment unit is used to dynamically collect monitoring indicators of the monitoring database server when data extraction is not completed, adjust various factors of the thread according to the dynamically collected monitoring indicators, and execute data extraction according to the various factors of the thread.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images