Dynamic adjustment method, device and equipment for database full-amount synchronization and medium

Abstract

The invention relates to the technical field of IT and software development, in particular to a dynamic adjustment method, device and equipment for database full-amount synchronization and a medium. The method comprises the following steps: acquiring a data table in full-amount synchronization in a database; according to the storage time consumption of each data table, adjusting the data volume of a single storage batch and the data volume of an extraction batch; adjusting an extraction thread according to the size of the memory used by the data temporary storage queue and the relationship between the queue length and a threshold value; and according to the warehousing failure rate of the data table, adjusting the data volume of a single warehousing batch and the number of concurrent warehousing threads. According to the method and the device, the storage time consumption of the data is monitored, and the single-batch extraction and storage data volume is adjusted, so that the problems of long-time blockage waiting, database connection write-in timeout and the like in batch storage are avoided. Meanwhile, the memory usage amount of the data temporary storage queue and the length of the data temporary storage queue are monitored, and memory overflow and synchronization lagging are prevented. And the concurrent warehousing thread count and the single-batch warehousing data volume are adjusted according to the warehousing failure rate, so that the network and database pressure is relieved.

Description

Technical Field

[0001] The present invention relates to the field of IT and software development technology, and in particular to a dynamic adjustment method, device, equipment and medium for full synchronization of a database. Background Art

[0002] When using the database synchronization function to synchronize data in various resource pools, the stability and accuracy of full synchronization have an important impact on the deployment progress of the resource pool and subsequent online operations. Due to differences in network quality, hard disk type, database performance, etc. of each resource pool, the performance of full synchronization will also be different.

[0003] Currently, the full synchronization parameters of the data synchronization component, such as batch size, number of warehousing threads, number of concurrent tables, etc., are all configured as preset values, which will not change during the synchronization process and cannot be dynamically adjusted according to changes in the resource pool environment. At the same time, full synchronization is usually multi-table parallel, and the synchronization efficiency of each table will also be affected by other tables. The current data synchronization component uses common parameters for all tables, and cannot adjust the synchronization parameters for a single table. Summary of the invention

[0004] In view of this, the present invention provides a method, device, equipment and medium for dynamic adjustment of full synchronization of a database to solve the problem of lack of adjustment of full synchronization parameters of a data synchronization component in the prior art.

[0005] In a first aspect, the present invention provides a dynamic adjustment method for full synchronization of a database. The task objects of full synchronization of the database include an extraction thread pool, an warehousing thread pool and a data temporary storage queue. The extraction thread pool is used to concurrently extract multiple batches of data from a data table through multiple extraction threads and put them into the data temporary storage queue. The warehousing thread pool is used to concurrently read multiple batches of data from the data temporary storage queue through multiple warehousing threads. The method includes: obtaining a data table in full synchronization in the database; counting the warehousing time of each data table, and adjusting the data volume of a single warehousing batch and the data volume of a single extraction batch according to the warehousing time; adjusting the extraction thread according to the relationship between the memory size used by the data temporary storage queue of the data table and the queue length and the threshold of the data temporary storage queue; counting the warehousing failure rate of each data table, and adjusting the data volume of a single warehousing batch and the number of concurrent warehousing threads according to the warehousing failure rate.

[0006] In the present invention, by monitoring the time consumption of data entry, adjusting the amount of data extracted and entered in a single batch, and avoiding problems such as long-term blocking waiting for batch entry and database connection write timeout. At the same time, the memory usage of the data temporary storage queue and the length of the data temporary storage queue are monitored to prevent memory overflow and synchronization jam. In addition, the number of concurrent entry threads and the amount of single batch entry data can be adjusted according to the entry failure rate, thereby reducing network and database pressure, avoiding frequent errors, and ensuring the stability of synchronization. Therefore, in the full synchronization scenario, this method can dynamically adjust parameters such as the extraction batch size, the entry batch size, and the number of concurrent threads according to changes in the real-time operating environment, use resources more reasonably, and effectively reduce the occurrence of common problems such as database connection timeout disconnection, jamming caused by concurrent synchronization of large tables, and memory overflow, avoid frequent errors, and improve synchronization efficiency and stability.

[0007] In an optional implementation, the warehousing time of each data table is counted, and the data volume of a single warehousing batch and a single extracted batch is adjusted according to the warehousing time, including: counting the warehousing time of each batch of each data table, and calculating the average warehousing time of each data table; when the average warehousing time is greater than or equal to the time threshold, adjusting the data volume of a single warehousing batch and the data volume of a single extracted batch; when the average warehousing time is less than the time threshold, obtaining the number of data tables in full synchronization; when the number is greater than a preset number, determining the total warehousing time according to the average warehousing time of each data table; and adjusting the data volume of a single warehousing batch and the data volume of a single extracted batch according to the relationship between the average warehousing time and the total warehousing time.

[0008] In the present invention, by monitoring the storage time of data and calculating the average storage time, when the average storage time is greater than or equal to the time threshold, the amount of data extracted and stored in a single batch is adjusted to avoid problems such as long-term blocking waiting for batch storage and database connection write timeout. When the average storage time is less than the time threshold, the average storage time is further compared with the total storage time, and the data table with a longer storage time is determined, and the amount of data extracted and stored is further adjusted to ensure the balance of storage time.

[0009] In an optional implementation, when the average storage time is greater than or equal to the time threshold, the adjusted data volume of a single storage batch and the adjusted data volume of a single extraction batch are respectively expressed by the following formulas:

[0010]

[0011] In the formula, newPushSize and oldPushSize represent the data volume of a single incoming batch before and after the adjustment, newPullSize and oldPullSize represent the data volume of a single extracted batch, threshold represents the time consumption threshold, and avgCost represents the average incoming time consumption;

[0012] According to the relationship between the average warehousing time and the total warehousing time, the adjusted data volume of a single warehousing batch and the adjusted data volume of a single extraction batch are expressed by the following formulas:

[0013]

[0014] In the formula, totalCost represents the total warehousing time.

[0015] In the present invention, the data volume of a single warehousing batch and the data volume of a single extraction batch are adjusted by using the above formula, so that the adjusted warehousing time is more balanced.

[0016] In an optional implementation, obtaining data tables in a database that are in full synchronization includes: placing information of all data tables in the database in a queue to be synchronized; extracting a corresponding number of data tables from the queue to be synchronized according to a preset number of full concurrent tables, and creating a full synchronization task; when the full synchronization task of any data table is completed, extracting the next data table from the queue to be synchronized to create a full synchronization task; determining the data table in full synchronization according to the data table that is executing the full synchronization task.

[0017] In the present invention, when full synchronization is performed, a certain number of data tables are extracted each time and performed concurrently, thereby improving the efficiency of warehousing. At the same time, when a data table completes full synchronization, a new data table is extracted to create a full synchronization task, thereby ensuring that a certain number of data tables are always fully synchronized.

[0018] In an optional implementation, the threshold of the data temporary storage queue includes a maximum memory usage threshold and a critical threshold of a single data temporary storage queue, and the extraction thread is adjusted according to the relationship between the memory size used by the data temporary storage queue of the data table and the queue length and the threshold of the data temporary storage queue, including: calculating the memory usage of full synchronization according to the memory size used by the data temporary storage queue of a single data table; when the memory usage is greater than the maximum memory usage threshold, pausing the extraction of the next data table from the queue to be synchronized to create a full synchronization task, and pausing the extraction thread of the data table with the largest memory used by the data temporary storage queue; adjusting the extraction thread according to the relationship between the length value of the data temporary storage queue of a single data table and the critical threshold of the single data temporary storage queue.

[0019] In the present invention, by monitoring the overall memory usage of the data temporary storage queue and taking measures such as suspending extraction when the upper limit is reached, the amount of data synchronized concurrently is effectively controlled to prevent memory overflow and synchronization jams.

[0020] In an optional implementation, the critical threshold of a single data temporary storage queue includes an upper critical threshold and a lower critical threshold. According to the relationship between the length value of the data temporary storage queue of a single data table and the critical threshold of the single data temporary storage queue, the extraction thread is adjusted, including: calculating the sliding average of the queue length at the current moment according to the length value of the data temporary storage queue of the single data table; when the sliding average of the queue length is greater than the upper critical threshold, reducing the number of concurrently executed extraction threads; when the sliding average of the queue length is less than the lower critical threshold, increasing the amount of data processed by a single warehousing batch.

[0021] In the present invention, the data flow rate of each data table is adjusted according to the length of the data temporary storage queue of a single data table. When the queue is long, the extraction rate is reduced to relieve the pressure of resource use; when the queue is short, the storage rate is increased, thereby making full use of resources.

[0022] In an optional implementation, when the data volume of a single storage batch and the number of concurrent storage threads are adjusted according to the storage failure rate, the adjusted data volume of the storage batch and the adjusted number of concurrent storage threads are respectively expressed by the following formulas:

[0023]

[0024] Where failRate∈(0,1) represents the failure rate of storage, newPushSize and oldPushSize represent the data size of a single storage batch before and after adjustment, and newPushWorkers and oldPushWorkers represent the number of concurrent storage threads before and after adjustment.

[0025] In the present invention, the data volume of a single storage batch of a data table with a high failure rate is reduced according to the above formula, and the number of concurrent storage threads is reduced, thereby alleviating the pressure on the network and the database and avoiding frequent errors.

[0026] In the second aspect, the present invention provides a dynamic adjustment device for full synchronization of a database. The task objects of the full synchronization of the database include an extraction thread pool, an warehousing thread pool and a data temporary storage queue. The extraction thread pool is used to concurrently extract multiple batches of data from a data table through multiple threads and put them into the data temporary storage queue. The warehousing thread pool is used to concurrently extract multiple batches of data from the data temporary storage queue through multiple threads. The device includes: a synchronization determination module, used to obtain a data table in full synchronization in the database; a first adjustment module, used to count the warehousing time of each data table, and adjust the data volume of a single warehousing batch and / or the data volume of a single extraction batch according to the warehousing time; a second adjustment module, used to adjust the extraction thread according to the relationship between the memory size used by the data temporary storage queue of the data table and the queue length and the threshold of the data temporary storage queue; a third adjustment module, used to count the warehousing failure rate of each data table, and adjust the data volume of a single warehousing batch and the number of concurrent warehousing threads according to the warehousing failure rate.

[0027] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the dynamic adjustment method for full synchronization of a database according to the first aspect or any corresponding embodiment thereof.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the dynamic adjustment method for full synchronization of a database according to the first aspect or any corresponding embodiment thereof.

[0029] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for dynamic adjustment of full synchronization of a database according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a flowchart of a dynamic adjustment method for full database synchronization according to an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of the composition of a full synchronization task according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a process for adjusting parameters according to the storage time consumption according to an embodiment of the present invention;

[0034] Figure 4 is a schematic diagram of a process flow for adjusting overall queue memory according to an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of a process for adjusting the queue length of a single table according to an embodiment of the present invention;

[0036] Figure 6 is a structural block diagram of a dynamic adjustment device for full database synchronization according to an embodiment of the present invention;

[0037] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0039] According to an embodiment of the present invention, an embodiment of a dynamic adjustment method for full synchronization of a database is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] In this embodiment, a dynamic adjustment method for full database synchronization is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of a dynamic adjustment method for full database synchronization according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0041] Step S101, obtaining a data table in full synchronization in the database.

[0042] Specifically, the full database synchronization can be implemented by the database data synchronization component, which can synchronize all data in the specified source data table to the corresponding target data table during the full synchronization. Figure 2As shown, the full synchronization task object of each data table is mainly composed of an extraction thread pool, an warehousing thread pool, and a data temporary storage queue. Among them, the extraction thread pool is used to manage several extraction threads, divide the data into multiple batches according to the primary key of the source data table, and multiple extraction threads concurrently extract different batches of data from the data source table and put them into the data temporary storage queue. The data temporary storage queue is used to receive the data obtained by the extraction thread and provide it to the warehousing thread for consumption. The warehousing thread pool is used to manage several warehousing threads, and multiple batches of data are concurrently read from the data temporary storage queue each time through multiple warehousing threads, and inserted into the target data table.

[0043] Since the database may include multiple data tables, when using the data synchronization component to perform the full synchronization function of the database, if the full synchronization task is performed on one data table each time, it may take a long time; if the full synchronization task is performed on all data tables in the database each time, the mutual influence of the synchronization efficiency between the data tables may slow down the execution of the full synchronization task. Therefore, a certain number of data tables in the database can be selected each time to perform the full synchronization task in parallel. Here, the data table in full synchronization can be a data table in the database that is currently performing full synchronization.

[0044] Step S102, counting the storage time of each data table, and adjusting the data volume of a single storage batch and the data volume of a single extraction batch according to the storage time.

[0045] Specifically, multiple extraction threads in the extraction thread pool concurrently extract different batches of data from the data table, and multiple storage threads in the storage thread pool concurrently read different batches of data from the data temporary storage queue. Therefore, the extraction time and storage time of a single batch can be monitored, wherein the extraction time of a single batch refers to the time from the extraction thread initiating an extraction request to the source data table to the time when all the data of the batch is inserted into the data temporary storage queue. The storage time of a single batch refers to the time from the time when the batch of data is extracted by the storage thread in the data temporary storage queue to the time when all the data of the batch is inserted into the target data table. In the process of performing full synchronization, the storage time of a single batch of each data table can be counted at intervals, and the data volume of a single storage batch and a single extraction batch can be appropriately adjusted based on the statistical storage time to avoid long-term blocking and waiting of data.

[0046] Step S103, adjusting the extraction thread according to the relationship between the memory size and queue length used by the data temporary storage queue of the data table and the threshold of the data temporary storage queue. Specifically, in the process of executing the full synchronization task of the data table, the memory size and queue length used by the data temporary storage queue are monitored, and the monitoring result is compared with the preset threshold of the data temporary storage queue to determine whether to adjust the extraction thread, thereby avoiding memory overflow and achieving full utilization of memory resources.

[0047] Step S104, count the warehousing failure rate of each data table, and adjust the data volume and concurrent warehousing threads of a single warehousing batch according to the warehousing failure rate. Specifically, if the full synchronization of the database is applied in the full synchronization scenario of the cloud computer resource pool, cross-computer room and cross-regional data transmission is required, which is easily affected by the network quality during transmission, resulting in failure of warehousing of some batches of data. For example, network jitter causes the database connection to be disconnected, network delay causes the data transmission time to exceed the timeout time set by the target library, and so on.

[0048] After a batch fails to be loaded into the warehouse, the system will initiate a retry and try to load it into the warehouse again. If it still fails, the primary key range of the batch data will be recorded. After the full synchronization is completed, the user can initiate a supplementary upload for the data table with the failed batch. Specifically, the data of these batches will be re-extracted for storage based on the primary key range of the previously recorded failed batch. If the storage failure rate is high, it can be considered that the network quality is not good or the database writing pressure is too high. At this time, the data volume of a single storage batch and the number of concurrent storage threads can be appropriately adjusted to reduce the pressure on the network and database and avoid frequent errors.

[0049] The dynamic adjustment method for full database synchronization provided by the embodiment of the present invention adjusts the amount of data extracted and stored in a single batch by monitoring the time consumption of data storage, thereby avoiding problems such as long-term blocking waiting for batch storage and database connection write timeout. At the same time, the memory usage of the data temporary storage queue and the length of the data temporary storage queue are monitored to prevent memory overflow and synchronization jams. In addition, the number of concurrent storage threads and the amount of single batch storage data can be adjusted according to the storage failure rate, thereby reducing the pressure on the network and database, avoiding frequent errors, and ensuring the stability of synchronization. Therefore, in the full synchronization scenario, the method can dynamically adjust the extraction batch size, storage batch size, number of concurrent threads and other parameters according to the changes in the real-time operating environment, use resources more reasonably, and effectively reduce the occurrence of common problems such as database connection timeout disconnection, jams caused by concurrent synchronization of large tables, and memory overflow, avoid frequent errors, and improve synchronization efficiency and stability.

[0050] In this embodiment, a method for dynamic adjustment of full database synchronization is provided, and the process includes the following steps:

[0051] Step S201, obtaining a data table in full synchronization in the database.

[0052] Specifically, the above step S201 includes:

[0053] Step S2011, placing information of all data tables in the database into a queue to be synchronized.

[0054] Step S2012: extract a corresponding number of data tables from the queue to be synchronized according to the preset number of full concurrent tables, and create a full synchronization task.

[0055] Step S2013: When the full synchronization task of any data table is completed, the next data table is extracted from the queue to be synchronized to create a full synchronization task.

[0056] Step S2014, determining the data table in full synchronization according to the data table that is executing the full synchronization task.

[0057] Specifically, after the full synchronization of the database begins, the information of all data tables will be placed in a queue to be synchronized. The system extracts the corresponding number of data tables from the queue to be synchronized according to the preset number of full concurrent tables, creates full synchronization tasks for these data tables, and then synchronizes them in parallel. After the synchronization of a data table is completed, the next data table will be extracted from the queue to be synchronized. Among them, when performing full synchronization on each data table, relevant parameters can be monitored and dynamically adjusted according to the following steps S202 to S204.

[0058] Step S202, counting the storage time of each data table, and adjusting the data volume of a single storage batch and / or the data volume of a single extraction batch according to the storage time.

[0059] Specifically, the above step S202 includes:

[0060] Step S2021, count the warehousing time of each batch of each data table, and calculate the average warehousing time of each data table; wherein, for each data table, when a batch is completed into the warehouse, its warehousing time (i.e., the warehousing time of a single batch) can be monitored, and then at intervals, the average warehousing time of multiple recent batches of each data table can be counted according to the monitored warehousing time. For example, the average warehousing time of the most recent 10 batches can be counted. If the number of batches that have completed warehousing is less than 10, the average warehousing time of the completed batches is calculated to obtain the corresponding average warehousing time, which can be used as the average warehousing time of the corresponding data table.

[0061] Step S2022, when the average storage time is greater than or equal to the time threshold, adjust the data volume of a single storage batch and the data volume of a single extraction batch; specifically, the time threshold can be predetermined according to actual conditions, for example, the time threshold can be set to 60 seconds. When the average storage time of a data table is greater than or equal to the time threshold, the data volume of a single storage batch of the data table can be reduced, and the data volume of a single extraction batch can be adjusted accordingly. The adjusted data volume of a single storage batch and the adjusted data volume of a single extraction batch are respectively expressed by the following formulas:

[0062]

[0063] In the formula, newPushSize and oldPushSize represent the data size of a single incoming batch before and after adjustment, newPullSize and oldPullSize represent the data size of a single extracted batch, threshold represents the time consumption threshold, and avgCost represents the average incoming time consumption.

[0064] Step S2023, when the average storage time is less than the time threshold, the number of data tables in full synchronization is obtained.

[0065] Step S2024, when the number is greater than 2, the total storage time is determined according to the average storage time of each data table.

[0066] Specifically, when the average warehousing time of each data table is counted and it is determined that the average warehousing time of each data table is less than the time threshold, the number of data tables in full synchronization is obtained. If the number is greater than two, the total warehousing time is further counted. The total warehousing time is the sum of the average warehousing time of all data tables.

[0067] Step S2025, according to the relationship between the average warehousing time and the total warehousing time, adjust the data volume of a single warehousing batch and the data volume of a single extracted batch. Specifically, when the average warehousing time of each data table is less than the time threshold, the average warehousing time and the total warehousing time of each data table can be further compared to determine whether there are data tables with long warehousing time. For example, when the average warehousing time of a data table is greater than half of the total warehousing time, it means that the warehousing time of the data table is long. At this time, the data volume of a single warehousing batch of the data table can be reduced, and the data volume of a single extracted batch can be adjusted accordingly. Among them, the adjusted data volume of a single warehousing batch and the adjusted data volume of a single extracted batch are respectively expressed by the following formulas:

[0068]

[0069] In the formula, totalCost represents the total warehousing time.

[0070] Step S203, adjusting the extraction thread according to the relationship between the memory size used by the temporary data storage queue of the data table and the queue length and the threshold of the temporary data storage queue;

[0071] Specifically, the above step S203 includes:

[0072] Step S2031, calculate the memory usage of full synchronization according to the memory size used by the temporary data storage queue of a single data table; specifically, the memory usage of full synchronization is the sum of the memory sizes used by the temporary data storage queues of each data table. For example, the memory sizes used by the temporary data storage queues of the data table that is executing the full synchronization task can be counted at preset time intervals, and then added up to obtain the memory usage of full synchronization at the current moment. Among them, the memory size used by the temporary data storage queue of the data table can be determined according to the product of the memory size used by the single data extracted by the extraction thread and the queue length.

[0073] Step S2032, when the memory usage is greater than the maximum memory usage threshold, suspend the extraction of the next data table from the queue to be synchronized to create a full synchronization task, and suspend the extraction thread of the data table with the largest memory used by the data temporary storage queue; specifically, the maximum memory usage threshold is the maximum memory usage allowed for full synchronization. When the threshold is exceeded, there may be a risk of memory overflow. Compare the calculated memory usage of the full synchronization at the current moment with the maximum memory usage threshold. If the memory usage is greater than the maximum memory usage threshold, suspend the extraction of the next data table from the queue to be synchronized to create a full synchronization task, and suspend the extraction thread of the data table with the largest memory used by the data temporary storage queue. At the same time, after the pause, the memory usage of the full synchronization can be calculated once every preset time interval, and compared with the maximum memory usage threshold. If the memory usage drops to a certain proportion of the maximum memory usage threshold, such as 60%, the above-mentioned suspended task can be resumed, that is, it is allowed to continue to obtain table information from the queue to be synchronized. At the same time, the suspended extraction thread is resumed to allow it to continue working.

[0074] Step S2033, according to the relationship between the length value of the data temporary storage queue of a single data table and the critical threshold value of the single data temporary storage queue, the extraction thread is adjusted. Specifically, for each data table data temporary storage queue, there is a maximum length value, and the upper critical threshold value and the lower critical threshold value of each data temporary storage queue are calculated according to the maximum length value, wherein the upper critical threshold value = maximum length value × 0.8, and the lower critical value = maximum length value × 0.2.

[0075] Specifically, the above step S2033 includes:

[0076] Step a1, calculate the sliding average of the queue length at the current moment according to the length value of the temporary data storage queue of a single data table; specifically, the sliding average of the queue length can be calculated once every preset time interval after the first batch of data in the data table is successfully stored. The sliding average of the queue length can be calculated using the following formula:

[0077]

[0078] In the formula, len t Indicates the queue length obtained at the tth moment, avg t represents the sliding average of the queue length at the tth moment, and β∈(0,1) is the weight parameter.

[0079] Step a2: when the sliding average value of the queue length is greater than the upper critical threshold, the number of concurrently executed extraction threads is reduced; specifically, when the sliding average value of the queue length is greater than the upper critical threshold, it is determined that there is too much backlog data in the queue, so the number of concurrently executed extraction threads can be reduced. The calculation method of the number of concurrent extraction threads after reduction is as follows:

[0080]

[0081] Where pullWorkers is the adjusted number of concurrent extraction threads, pushWorkers is the current number of concurrent storage threads, pullCost and pushCost are the average extraction time and average storage time of the last 10 batches, respectively, and pullSize and pushSize are the data size of a single extraction batch and the data size of a single storage batch, respectively.

[0082] Step a3: when the sliding average of the queue length is less than the lower critical threshold, the data volume processed by a single incoming batch is increased. Specifically, when the sliding average of the queue length is less than the lower critical threshold, it is determined that the queue is relatively idle and the data table synchronization speed is fast, so the data volume processed by a single incoming batch can be increased. If the adjusted single incoming batch data volume is greater than the extracted batch data volume, the data volume of the single extracted batch also needs to be increased to be consistent with the incoming batch data volume.

[0083] Step S204, count the failure rate of each data table, and adjust the data volume and concurrent threads of a single batch of data entering the warehouse according to the failure rate of data entering the warehouse. When counting the failure frequency, the data can be counted once every preset time interval, and the statistics can be determined based on the number of failures of multiple batches that have been recently entered into the warehouse. For example, if 5 out of the 10 batches that have been recently entered into the warehouse fail to enter the warehouse, the failure rate of data entering the warehouse is 5 / 10=50%. Specifically, when the failure rate is determined to be high, the following formula can be used to calculate the adjusted data volume of the batch entering the warehouse and the adjusted number of concurrent threads of data entering the warehouse:

[0084]

[0085] Where failRate∈(0,1) represents the failure rate of storage, newPushSize and oldPushSize represent the data size of a single storage batch before and after adjustment, and newPushWorkers and oldPushWorkers represent the number of concurrent storage threads before and after adjustment.

[0086] The dynamic adjustment method for full database synchronization provided by the present invention adjusts the amount of single batch storage data and the amount of single batch extraction data when the data batch storage time consumption in the recent period is high, effectively reducing the long-term blocking waiting of batch storage and the occurrence of database write timeout. By monitoring the overall memory usage of the data temporary storage queue and taking measures such as suspending extraction when the upper limit is reached, the amount of concurrently synchronized data is effectively controlled to prevent memory overflow and synchronization jam. According to the length of the data temporary storage queue of a single table, the data flow rate of each table is adjusted. When the queue is long, the extraction rate is reduced to relieve the pressure of resource use; when the queue is short, the storage rate is increased to make full use of resources. At the same time, the amount of single batch storage data and the number of storage threads can be adjusted according to the storage failure rate, and the failed primary key range can be recorded, thereby reducing the pressure on the network and database, avoiding frequent errors, and ensuring the stability of synchronization. When the network quality and database performance are poor, the number of exceptions thrown is reduced, and the synchronization stability is improved. At the same time, the method facilitates subsequent retransmission by recording the primary key range of failed storage.

[0087] The dynamic adjustment method for full database synchronization provided by the present invention can be used in the "one city, one pool" deployment scenario of Tianyi Cloud Computer. When the database is initialized, the full amount of data in the national region is synchronized to the resource pool. The operating environment of resource pools in various places is uneven. The use of this method can make full synchronization flexibly adapt to environmental changes, while ensuring the synchronization speed and improving stability. This method is suitable for database synchronization with a large amount of data, which is prone to database connection timeout disconnection, large table concurrent synchronization leading to blockage, memory overflow, etc. The implementation of this method can effectively reduce the occurrence of these problems.

[0088] The dynamic adjustment method for full database synchronization provided by the present invention can improve the operation and maintenance automation of data synchronization components, especially in the scenario where the amount of synchronized data is large, problems such as high storage time consumption and large memory usage are prone to cause abnormalities, and operation and maintenance personnel often need to handle alarms and manually adjust synchronization parameters. However, this method can automatically adjust synchronization parameters according to changes in indicators during operation, reducing the workload of operation and maintenance personnel. At the same time, the use of this method can ensure the stability of full data synchronization and reduce program errors and storage failure rates.

[0089] As a specific application example of the present invention, the dynamic adjustment method of full database synchronization can be implemented by the following process:

[0090] Step 1: Determine the data table for full synchronization.

[0091] After the full synchronization of the database begins, the information of all data tables will be placed in a queue to be synchronized. The system extracts the corresponding number of data tables from the queue to be synchronized according to the preset number of full concurrent tables, creates full synchronization tasks for these data tables, and then synchronizes them in parallel. After the synchronization of a data table is completed, the next data table will be extracted from the queue to be synchronized.

[0092] In addition, before synchronization, you can specify the default number of extraction threads, single batch extraction data volume, storage thread number, and single batch storage data volume in the configuration file of the data synchronization component. The single batch data volume is in the number of records. For example, if the single batch extraction data volume is 2000, it means that the extraction thread reads 2000 records from the source data table each time; if the single batch storage data volume is 2000, it means that the storage thread reads 2000 records from the data temporary storage queue each time and inserts them into the target data table.

[0093] Step 2: Adjust the batch size based on the time it takes to enter the warehouse.

[0094] At regular intervals, the warehousing time of each data table is counted, and the data volume of a single warehousing batch and the data volume of a single extraction batch are appropriately adjusted. Among them, for each data table, two queues with a length of 10 can be used to save the extraction time and warehousing time of the batch respectively. When a batch of data in the data table is extracted or warehousing, the extraction time or warehousing time will be inserted into the corresponding queue. If the queue is full, remove the earliest queued data before inserting it, so that the queue can save the extraction time and warehousing time of the most recent 10 batches. When you need to calculate the average warehousing time, traverse the queue of warehousing time and make statistics. In addition, the data synchronization component can count the warehousing time of all synchronized data tables every 30 seconds, and adjust the single batch data volume appropriately according to the results, such as Figure 3 As shown in the figure, the specific process of adjusting the batch size according to the time consumption of warehousing is as follows:

[0095] Step 2.1, calculate the average warehousing time of the most recent 10 batches of each data table (if the number of batches that have completed warehousing is less than 10, calculate the average warehousing time of the completed batches), and record it as the average warehousing time of the data table.

[0096] Step 2.2: If the average storage time of the data table exceeds the storage time threshold of a single table (this value is preset by the user, such as 60 seconds), it is necessary to reduce the amount of data in a single storage batch of the data table and adjust the amount of data in a single extraction batch accordingly. The amount of data in a single batch can be adjusted according to the following formula:

[0097]

[0098] In the formula, newPushSize and oldPushSize are the single batch data volume before and after adjustment, newPullSize and oldPullSize are the single batch data volume before and after adjustment, threshold is the storage time threshold, and avgCost is the average storage time.

[0099] Step 2.3: If the number of tables being synchronized is greater than 2, add up the average storage time of all tables being synchronized to get the total storage time; otherwise, end the process.

[0100] Step 2.4: If the average storage time of a table is greater than half of the total storage time, you need to reduce the amount of data in a single storage batch of the table and adjust the amount of data in a single extraction batch accordingly. The amount of data in a single batch can be adjusted according to the following formula:

[0101]

[0102] In the formula, totalCost is the total time consumed for warehousing.

[0103] Step 3: Control the data flow rate according to the queue length.

[0104] Step 3.1, adjust according to the overall queue memory.

[0105] Specifically, the process of adjusting according to the overall queue memory is as follows: Figure 4 As shown. Users can set the maximum memory usage threshold allowed for full synchronization in the configuration file of the data synchronization component, which is recorded as maxMemory in MB. At regular intervals, the memory size queueSize used by the data temporary storage queue of each data table is calculated, and the sum is added to obtain the current memory usage useMemory of full synchronization. Among them, the memory size used by the data temporary storage queue of a single data table can be calculated by the following method:

[0106] After the full synchronization of the data table begins, the extraction thread starts to send requests to the source data table. At this time, the sizes of the first 100 records read from the data source table are counted, and their average value avgRecordSize is calculated as the memory size used by a single data. The memory size queueSize used by the data temporary queue = queue length * avgRecordSize.

[0107] If useMemory>maxMemory is found, do the following:

[0108] a) It is forbidden to obtain table information from the queue to be synchronized. Even if the data table currently being synchronized completes synchronization at a later time, it is not allowed to start synchronization of a new data table.

[0109] b) Find the data table corresponding to the data temporary storage queue with the largest queueSize, and let all extraction threads of this data table suspend work after obtaining the current batch of data.

[0110] When useMemory drops to a certain level (for example, to 60% of maxMemory), the above two restrictions a and b are lifted and normal synchronization is resumed, that is, table information is allowed to continue to be obtained from the queue to be synchronized. The suspended extraction thread is resumed to allow it to continue working.

[0111] Step 3.2, make adjustments based on the queue length of a single table.

[0112] Specifically, the process of adjusting the queue length of a single table is as follows: Figure 5 As shown. Each data temporary storage queue has a maximum length value maxLen, set the upper critical threshold = maxLen × 0.8, and the lower critical threshold = maxLen × 0.2. After the first batch of data in the data table is successfully stored, record the current data temporary storage queue length len0. Thereafter, every 10 seconds as a sampling time point, obtain the current queue length len t , and calculate the sliding average of the queue length avg at the current moment t , the formula is as follows:

[0113]

[0114] In the formula, len t Indicates the queue length obtained at the tth moment, avg t represents the sliding average of the queue length at the tth moment, and t represents the tth sampling time point. β∈(0,1) is the weight parameter. Regarding the value of β, avg t Roughly equal to the average of the queue length values ​​at the past 1 / (1-β) moments. For example, β = 0.9, then avg tApproximately equal to the average queue length of the most recent 10 sampling time points.

[0115] Compare the sliding average of queue length to the critical value:

[0116] If the average value is greater than the upper critical threshold, it is determined that there is too much backlog data in the queue, so the number of concurrently executed extraction threads will be reduced. The calculation method is as follows:

[0117]

[0118] In the formula, pullWorkers is the adjusted number of concurrent extraction threads, pushWorkers is the current number of concurrent storage threads, pullCost and pushCost are the average extraction time and average storage time of the last 10 batches, respectively, and pullSize and pushSize are the data volume of a single extraction batch and the data volume of a single storage batch, respectively. Among them, pullCost and pushCost can be calculated by traversing the extraction time-consuming queue and the storage time-consuming queue.

[0119] If the average value is less than the lower critical threshold, it is determined that the queue is relatively idle and the synchronization speed of the data table is fast, so the amount of data processed by a single incoming batch will be increased. At the same time, in order to make full use of the data temporary storage queue space, if the adjusted single incoming batch data volume is greater than the single extracted batch data volume, the single extracted batch data volume needs to be increased to make it consistent with the incoming batch data volume. Among them, when increasing the amount of data processed by a single incoming batch, the current single batch incoming data volume of the data table can be doubled. For example, the original incoming thread processes 2,000 data in each batch, and after adjustment, 4,000 data in each batch.

[0120] Step 4: Processing of data that failed to be stored.

[0121] When a batch of data fails to be stored, the storage thread will immediately try to re-insert the batch into the target data table. If it still fails, the primary key range of the batch data is saved to the failed primary key set failKeySet corresponding to the data table. A Boolean queue with a length of 10 can be used to record whether the batch is successfully stored. If the storage is successful, insert true into the queue; if the storage fails, insert false into the queue. If the queue is full, remove the value at the head of the queue first, and then insert a new value. In this way, the storage failure rate of the last 10 batches can be counted based on the values ​​in the queue. Whenever a batch fails to be stored, the failure rate is counted once. If the storage failure rate is high, for example 50%, that is, 5 of the last 10 batches have failed to be stored, it can be considered that the network quality is not very good or the database write pressure is too high. It is necessary to reduce the amount of data for a single storage batch of the table and reduce the number of concurrent storage threads, thereby reducing the pressure on the network and database and avoiding frequent errors. The calculation method is as follows:

[0122]

[0123] Where failRate∈(0,1) is the failure rate of storage, newPushSize and oldPushSize are the single batch storage data size before and after adjustment, newPushWorkers and oldPushWorkers are the number of concurrent storage threads before and after adjustment.

[0124] After the full synchronization is completed, users can initiate a supplementary upload for the table with failed batches. They can re-extract the data of these batches and store them in the database based on the primary key range set failKeySet of the previously recorded failed batches.

[0125] In this embodiment, a dynamic adjustment device for full synchronization of a database is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0126] This embodiment provides a dynamic adjustment device for full database synchronization. The task objects of full database synchronization include an extraction thread pool, an warehousing thread pool, and a data temporary storage queue. The extraction thread pool is used to concurrently extract multiple batches of data from a data table through multiple threads and put them into the data temporary storage queue. The warehousing thread pool is used to concurrently extract multiple batches of data from the data temporary storage queue through multiple threads. Figure 6 As shown, the device comprises:

[0127] A synchronization determination module 61 is used to obtain a data table in full synchronization in the database;

[0128] The first adjustment module 62 is used to count the storage time of each data table and adjust the data volume of a single storage batch and / or the data volume of a single extraction batch according to the storage time;

[0129] A second adjustment module 63, used for adjusting the extraction thread according to the relationship between the memory size used by the temporary data storage queue of the data table and the queue length and the threshold value of the temporary data storage queue;

[0130] The third adjustment module 64 is used to count the storage failure rate of each data table and adjust the data volume of a single storage batch and the number of concurrent storage threads according to the storage failure rate.

[0131] In an optional embodiment, the first adjustment module is specifically used to: count the warehousing time of each batch of each data table, and calculate the average warehousing time of each data table; when the average warehousing time is greater than or equal to the time threshold, adjust the data volume of a single warehousing batch and the data volume of a single extracted batch; when the average warehousing time is less than the time threshold, obtain the number of data tables in full synchronization; when the number is greater than the preset number, determine the total warehousing time according to the average warehousing time of each data table; according to the relationship between the average warehousing time and the total warehousing time, adjust the data volume of a single warehousing batch and the data volume of a single extracted batch.

[0132] In an optional implementation, when the average storage time is greater than or equal to the time threshold, the adjusted data volume of a single storage batch and the adjusted data volume of a single extraction batch are respectively expressed by the following formulas:

[0133]

[0134] In the formula, newPushSize and oldPushSize represent the data volume of a single incoming batch before and after the adjustment, newPullSize and oldPullSize represent the data volume of a single extracted batch, threshold represents the time consumption threshold, and avgCost represents the average incoming time consumption;

[0135] According to the relationship between the average warehousing time and the total warehousing time, the adjusted data volume of a single warehousing batch and the adjusted data volume of a single extraction batch are expressed by the following formulas:

[0136]

[0137] In the formula, totalCost represents the total warehousing time.

[0138] In an optional implementation, the synchronization determination module is specifically used to: place the information of all data tables in the database in a queue to be synchronized; extract a corresponding number of data tables from the queue to be synchronized according to a preset number of full concurrent tables, and create a full synchronization task; when the full synchronization task of any data table is completed, extract the next data table from the queue to be synchronized to create a full synchronization task; determine the data table that is in full synchronization based on the data table that is executing the full synchronization task.

[0139] In an optional implementation, the threshold of the data temporary storage queue includes a maximum memory usage threshold and a critical threshold of a single data temporary storage queue, and the second adjustment module includes: a calculation module, used to calculate the memory usage of full synchronization according to the memory size used by the data temporary storage queue of a single data table; a pause module, used to pause the extraction of the next data table from the queue to be synchronized to create a full synchronization task when the memory usage is greater than the maximum memory usage threshold, and pause the extraction thread of the data table with the largest memory used by the data temporary storage queue; an adjustment sub-module, used to adjust the extraction thread according to the relationship between the length value of the data temporary storage queue of a single data table and the critical threshold of the single data temporary storage queue.

[0140] In an optional embodiment, the critical threshold of a single data temporary storage queue includes an upper critical threshold and a lower critical threshold, and the adjustment submodule is specifically used to: calculate the sliding average of the queue length at the current moment according to the length value of the data temporary storage queue of a single data table; when the sliding average of the queue length is greater than the upper critical threshold, reduce the number of concurrently executed extraction threads; when the sliding average of the queue length is less than the lower critical threshold, increase the amount of data processed by a single warehousing batch.

[0141] In an optional implementation, when the data volume of a single storage batch and the number of concurrent storage threads are adjusted according to the storage failure rate, the adjusted data volume of the storage batch and the adjusted number of concurrent storage threads are respectively expressed by the following formulas:

[0142]

[0143] Where failRate∈(0,1) represents the failure rate of storage, newPushSize and oldPushSize represent the data size of a single storage batch before and after adjustment, and newPushWorkers and oldPushWorkers represent the number of concurrent storage threads before and after adjustment.

[0144] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0145] The embodiment of the present invention also provides a computer device having the above Figure 6 The dynamic adjustment device for full synchronization of the database shown.

[0146] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0147] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0148] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0149] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0151] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0152] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0153] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0154] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A dynamic adjustment method for full database synchronization, characterized in that: The task objects for full database synchronization include an extraction thread pool, an warehousing thread pool, and a data temporary storage queue. The extraction thread pool is used to concurrently extract multiple batches of data from a data table through multiple extraction threads and put them into the data temporary storage queue. The warehousing thread pool is used to concurrently read multiple batches of data from the data temporary storage queue through multiple warehousing threads. The method includes: Get the data tables in full synchronization in the database; Count the time consumed for warehousing each data table, and adjust the data volume of a single warehousing batch and the data volume of a single extraction batch according to the warehousing time consumption; Adjusting the extraction thread according to the relationship between the memory size used by the data temporary storage queue of the data table and the queue length and the threshold value of the data temporary storage queue; The storage failure rate of each data table is counted, and the data volume of a single storage batch and the number of concurrent storage threads are adjusted according to the storage failure rate.

2. The method according to claim 1, characterized in that Count the storage time of each data table, and adjust the data volume of a single storage batch and the data volume of a single extraction batch according to the storage time, including: Count the time it takes to enter the warehouse for each batch of each data table, and calculate the average time it takes to enter the warehouse for each data table; When the average storage time is greater than or equal to the time threshold, the data volume of a single storage batch and the data volume of a single extraction batch are adjusted; When the average storage time is less than the time threshold, the number of data tables in full synchronization is obtained; When the number is greater than the preset number, the total storage time is determined according to the average storage time of each data table; According to the relationship between the average warehousing time and the total warehousing time, the data volume of a single warehousing batch and the data volume of a single extraction batch are adjusted.

3. The method according to claim 2, characterized in that When the average storage time is greater than or equal to the time threshold, the adjusted data volume of a single storage batch and the adjusted data volume of a single extraction batch are respectively expressed by the following formulas: In the formula, newPushSize and oldPushSize represent the data volume of a single incoming batch before and after the adjustment, newPullSize and oldPullSize represent the data volume of a single extracted batch, threshold represents the time consumption threshold, and avgCost represents the average incoming time consumption; According to the relationship between the average storage time and the total storage time, the adjusted data volume of a single storage batch and the adjusted data volume of a single extraction batch are respectively expressed by the following formulas: In the formula, totalCost represents the total warehousing time.

4. The method according to claim 1, characterized in that Get the data tables in the database that are in full synchronization, including: Put the information of all data tables in the database into the queue to be synchronized; Extract a corresponding number of data tables from the queue to be synchronized according to the preset number of full concurrent tables, and create a full synchronization task; When the full synchronization task of any data table is completed, the next data table is extracted from the queue to be synchronized to create a full synchronization task; The data tables that are in full synchronization are determined based on the data tables that are performing full synchronization tasks.

5. The method according to claim 4, characterized in that The threshold of the data temporary storage queue includes a maximum memory usage threshold and a critical threshold of a single data temporary storage queue. The extraction thread is adjusted according to the relationship between the memory size and queue length used by the data temporary storage queue of the data table and the threshold of the data temporary storage queue, including: Calculate the memory usage of full synchronization based on the memory size used by the data temporary storage queue of a single data table; When the memory usage is greater than the maximum memory usage threshold, suspend the extraction of the next data table from the to-be-synchronized queue to create a full synchronization task, and suspend the extraction thread of the data table with the largest memory used by the data temporary storage queue; The extraction thread is adjusted according to the relationship between the length value of the data temporary storage queue of a single data table and the critical threshold value of the single data temporary storage queue.

6. The method according to claim 5, characterized in that The critical threshold of a single data temporary storage queue includes an upper critical threshold and a lower critical threshold. According to the relationship between the length value of the data temporary storage queue of a single data table and the critical threshold of the single data temporary storage queue, the extraction thread is adjusted, including: Calculate the sliding average of the queue length at the current moment based on the length of the data temporary storage queue of a single data table; When the sliding average of the queue length is greater than the upper critical threshold, the number of concurrently executed extraction threads is reduced; When the sliding average of the queue length is less than the lower critical threshold, increase the amount of data processed in a single incoming batch.

7. The method according to claim 1, characterized in that When the data volume of a single storage batch and the number of concurrent storage threads are adjusted according to the storage failure rate, the adjusted data volume of the storage batch and the adjusted number of concurrent storage threads are respectively expressed by the following formulas: Where failRate∈(0,1) represents the failure rate of storage, newPushSize and oldPushSize represent the data size of a single storage batch before and after adjustment, and newPushWorkers and oldPushWorkers represent the number of concurrent storage threads before and after adjustment.

8. A dynamic adjustment device for full database synchronization, characterized in that: The task objects for full database synchronization include an extraction thread pool, an warehousing thread pool, and a data temporary storage queue. The extraction thread pool is used to concurrently extract multiple batches of data from a data table through multiple threads and put them into the data temporary storage queue. The warehousing thread pool is used to concurrently extract multiple batches of data from the data temporary storage queue through multiple threads. The device includes: A synchronization determination module is used to obtain data tables in the database that are in full synchronization; A first adjustment module is used to count the storage time of each data table, and adjust the data volume of a single storage batch and / or the data volume of a single extraction batch according to the storage time; A second adjustment module, used for adjusting the extraction thread according to the relationship between the memory size used by the temporary data storage queue of the data table and the queue length and the threshold value of the temporary data storage queue; The third adjustment module is used to count the storage failure rate of each data table, and adjust the data volume of a single storage batch and the number of concurrent storage threads according to the storage failure rate.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the dynamic adjustment method for full synchronization of a database as described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the dynamic adjustment method for full synchronization of a database according to any one of claims 1 to 7.

Images