Method, apparatus and device for dynamic adjustment of synchronization parameters

Abstract

This invention provides a method, apparatus, and device for dynamically adjusting synchronization parameters to address the technical problem of not being able to dynamically adapt synchronization parameters of CDC (Cyber-Controlled Synchronization) jobs based on the execution environment. This invention collects historical CDC synchronization job information, including endpoint information, synchronization parameters, node resource information, and task performance metrics, as training sample data. It trains a machine learning prediction model for batch size. During CDC synchronization job scheduling, based on job information within a specified time interval, the trained prediction model predicts the optimal batch size parameter, dynamically setting the batch size parameter of the CDC synchronization job. This makes the batch size synchronization parameters more adaptable to actual business scenarios, improving the intelligence of synchronization configuration and the efficiency of synchronization.

Description

Technical Field

[0001] This invention relates to the fields of data communication and big data technology, and in particular to a method, apparatus and device for dynamic adjustment of synchronization parameters. Background Technology

[0002] In data processing and analysis scenarios, real-time data synchronization and high availability are important performance indicators. This requirement necessitates the timely acquisition and processing of data from different data sources, while ensuring data accuracy and consistency.

[0003] Change Data Capture (CDC) technology refers to the technique of capturing data changes in a database and is a key technology for achieving data synchronization. CDC technology is generally divided into query-based CDC and log-based CDC.

[0004] Flink is a distributed stream processing framework with high throughput, low latency, high performance, and support for exactly-once semantics, making it widely used. Flink CDC (Change Data Capture) is a distributed data synchronization solution based on the Flink stream processing framework. It allows changes to data in the source database to be captured and transmitted to the target database in real time for processing and analysis. This real-time data synchronization capability enables enterprises to process and respond to data almost in real time, thereby improving the accuracy and timeliness of business decisions.

[0005] Flink CDC leverages Flink's parallel computing capabilities and optimization strategies to achieve fast and efficient high-performance, low-latency data synchronization on distributed clusters. Flink CDC's real-time data synchronization capabilities enable near real-time synchronization between the source and target data sources, meeting the needs of applications requiring real-time data updates. Furthermore, Flink CDC offers flexible data processing capabilities, allowing for real-time transformation, filtering, aggregation, and computation of changing data from the main data source. This enables data cleaning, format conversion, and merging operations during data synchronization, satisfying diverse business requirements.

[0006] When performing data synchronization, a full data extraction is usually performed during the initial synchronization to ensure that the target system and the source system are completely consistent. Incremental extraction is then used to achieve real-time or periodic incremental data synchronization to maintain the real-time nature of the data.

[0007] Full load extraction refers to the process of retrieving all data from a data source. In full load extraction, the entire contents of the source data are read and copied to the target system for synchronization. Full load extraction is typically used during initial or periodic synchronization to ensure that the target system maintains consistency with the source data. Full load extraction can copy the complete dataset from the source to the target, regardless of whether the data has changed.

[0008] Incremental load extraction refers to the process of retrieving only the changed data from the data source. Incremental extraction monitors changes in the source data and copies only the changed portion to the target system for synchronization. Incremental extraction improves data synchronization efficiency and reduces the amount and time of data transmission. This method is suitable for situations where the source data changes frequently and can achieve real-time or near real-time data synchronization.

[0009] Currently, data synchronization solutions using CDC technology typically require manually setting configuration parameters such as batch size. The batch size parameter controls the amount of data processed at one time; it can be the number of data records or the actual size of the data (units such as KB, MB, GB, etc.). In batch processing mode, data is grouped according to a certain batch size and then processed in batches for synchronization.

[0010] Configuration parameters such as batch size have a significant impact on data synchronization efficiency. For example, a larger batch size can reduce the number of data reads and writes, thereby reducing processing overhead and improving data processing efficiency, but it will also increase memory and network load. A smaller batch size can reduce processing latency, but may lead to frequent data read and write operations.

[0011] Current solutions using CDC technology for distributed data synchronization cannot comprehensively consider factors such as actual needs, system resources, data characteristics, and latency requirements, nor can they dynamically optimize configuration parameters that affect synchronization efficiency, such as batch size, resulting in low data synchronization efficiency. Summary of the Invention

[0012] In view of this, the present invention provides a method, apparatus and device for dynamically adjusting synchronization parameters to solve the technical problem of not being able to dynamically adapt the synchronization parameters of CDC synchronization tasks according to the execution environment.

[0013] Based on one aspect of the embodiments of the present invention, the present invention provides a method for dynamically adjusting synchronization parameters, the method comprising:

[0014] The monitoring and submission of data changes to the distributed stream processing framework cluster captures CDC synchronization job tasks. The job information of historical CDC synchronization jobs is collected to build a training sample set. The job information includes: information of both endpoints, synchronization parameters, synchronization node resource information and task performance indicators.

[0015] The regression prediction model is trained using a training sample set. The input feature values ​​of the regression prediction model are constructed from job information excluding batch size, and the output is the label value of the batch size.

[0016] Before scheduling the CDC synchronization job to be scheduled, calculate the expected data throughput and expected data processing latency based on the historical job information of the CDC synchronization job to be scheduled within a specified time range before the current time.

[0017] Using the job information of the CDC synchronization job to be scheduled, excluding data throughput, data processing latency, and batch size, other job information, as well as expected data throughput and expected data processing latency, are used to construct feature values. The feature values ​​are then input into the regression prediction model, and the batch size output by the regression prediction model is used as the batch size of the CDC synchronization job to be scheduled. The CDC synchronization job to be scheduled is then executed.

[0018] Furthermore, the endpoint information refers to the relevant information of the data source and data destination of the CDC synchronization task, including: the database types of the source and destination, the host IP addresses of the databases at both ends, the data source table, and the data destination table;

[0019] The synchronization parameters refer to the synchronization parameters configured for the CDC synchronization job task, including: fragment size, parallelism, and batch size;

[0020] The synchronization node resource information refers to the resource usage information of the node executing the CDC synchronization task, including: the usage of CPU, memory, and network bandwidth resources of the executing node.

[0021] The task performance metrics refer to the performance indicators that reflect the performance of the nodes performing the CDC synchronization task, including: data processing latency, data throughput, data distribution balance, and average size of each data item.

[0022] Furthermore, for different execution nodes in the distributed stream processing framework cluster, the job information of the historical CDC synchronous job tasks of each execution node is collected, and training sample sets for each execution node are constructed respectively.

[0023] The regression prediction model for each execution node is trained using the training sample set of each execution node.

[0024] When scheduling CDC synchronization jobs, the expected data throughput and expected data processing latency are calculated for each execution node in the cluster, and feature values ​​are constructed.

[0025] Each execution node uses its own regression prediction model to predict the batch size of each execution node, and schedules the scheduled CDC synchronization job task based on the predicted batch size.

[0026] Furthermore, the method for calculating the expected data throughput and expected data processing latency for each execution node within the cluster is as follows:

[0027] Filter out the N smallest data processing delay records for the CDC synchronization job on the execution node within a specified time range before the current time; where N is a positive integer greater than 1.

[0028] Filter out the job information records corresponding to the top M largest data throughputs of the CDC synchronization job on the execution node within a specified time range before the current time; where M is a positive integer greater than 1;

[0029] Calculate the average data processing latency and data throughput of the selected job information records, and remove job information records corresponding to outliers based on the standard deviation;

[0030] For each execution node in the cluster, the average of the data processing latency and data throughput of the remaining job information records after removing outliers is determined as the expected data processing latency and expected data throughput of that execution node.

[0031] Furthermore, the method also includes:

[0032] The training sample set is enriched by continuously collecting historical CDC synchronous task information, and the regression prediction model is optimized through incremental training.

[0033] Furthermore, the regression prediction model is a linear regression model, a multinomial regression model, a ridge regression model, a decision tree regression model, a support vector regression (SVR) model, or a Lasso regression model.

[0034] Based on another aspect of the embodiments of the present invention, the present invention also provides a device for dynamically adjusting synchronization parameters. The device provided by the present invention can be implemented in software, hardware, or a combination of both. When implemented as a software module, the program code of the software module is loaded into the storage medium of the device, and the processor reads and executes the program code from the storage medium. The device includes:

[0035] The sample set construction module is used to monitor data changes submitted to the distributed stream processing framework cluster, capture CDC synchronization job tasks, collect job information of historical CDC synchronization jobs, and construct a training sample set. The job information includes: end point information, synchronization parameters, synchronization node resource information, and task performance indicators.

[0036] The model training module is used to train a regression prediction model using a training sample set. The input feature values ​​of the regression prediction model are constructed from job information excluding batch size, and the output is the label value of the batch size.

[0037] The expected metrics calculation module is used to calculate the expected data throughput and expected data processing latency of the CDC synchronization job to be scheduled before scheduling the CDC synchronization job to be scheduled, based on the historical job information of the CDC synchronization job to be scheduled within a specified time range before the current time.

[0038] The scheduling module is used to construct feature values ​​from the job information of the CDC synchronization job to be scheduled, excluding data throughput, data processing latency, and batch size, as well as the expected data throughput and expected data processing latency. The feature values ​​are input into the regression prediction model, and the batch size output by the regression prediction model is used as the batch size of the CDC synchronization job to be scheduled. The CDC synchronization job to be scheduled is then executed.

[0039] Furthermore, the sample set construction module collects the job information of the historical CDC synchronous job tasks of different execution nodes in the distributed stream processing framework cluster, and constructs the training sample set of each execution node respectively.

[0040] The model training module uses the training sample set of each execution node to train the regression prediction model of each execution node;

[0041] When scheduling CDC synchronization tasks, the expected indicator calculation module calculates the expected data throughput and expected data processing latency for each execution node in the cluster and constructs feature values.

[0042] The scheduling module uses the respective regression prediction model of each execution node to predict the batch size of each execution node, and schedules the scheduled CDC synchronization job task to be executed based on the predicted batch size.

[0043] Can this invention make the batch size synchronization parameter more suitable for actual business scenarios, and improve the intelligence and efficiency of synchronization configuration? Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained from these drawings of the embodiments of the present invention.

[0045] Figure 1 This is a flowchart illustrating the steps of a method for dynamically adjusting synchronization parameters according to an embodiment of the present invention.

[0046] Figure 2 This is a flowchart illustrating the steps of a method for calculating expected data throughput and expected data processing latency in one embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram of an electronic device structure for implementing the synchronization parameter dynamic adjustment method provided by the present invention, according to an embodiment of the present invention. Detailed Implementation

[0048] The terminology used in this embodiment of the invention is for the purpose of describing specific embodiments only and is not intended to limit the embodiments of the invention. The singular forms “a,” “the,” and “the” used in this embodiment are also intended to include the plural forms unless the context clearly indicates otherwise. It should be understood that although the terms first, second, third, etc., may be used to describe various information in this embodiment, this information should not be limited to these terms. These terms are only used to distinguish similar information, entities, or steps, and not to describe a specific order or sequence. For example, without departing from the scope of this embodiment, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Furthermore, the word “if” used can be interpreted as “when…”, “when…”, or “in response to a determination.” The term “and / or” in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. And, in the description of this invention, unless otherwise stated, “multiple” means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0049] When using CDC technology for data synchronization, it is necessary to set various configuration parameters such as job shard size, parallelism, and batch size. Improper settings of these parameters can negatively impact data synchronization performance. To clarify the relationship between these configuration parameters and data synchronization performance, the functions and roles of these parameters are briefly introduced below.

[0050] Chunk Size: Chunk size controls the size of data chunks used to divide large datasets. During data synchronization, large datasets are divided into smaller chunks to facilitate parallel processing and distributed computing. Smaller chunk sizes improve parallelism, allowing multiple jobs to process different chunks simultaneously, thus increasing overall data processing speed.

[0051] Parallelism: Parallelism controls the ability to execute tasks or process data simultaneously. In distributed computing frameworks, increasing parallelism allows for the utilization of more computing resources, thereby improving data processing speed and throughput. Parallelism can be understood as the number of tasks or data processes executed concurrently. Higher parallelism can fully utilize the cluster's computing resources, but resource scheduling and load balancing issues also need to be considered.

[0052] Batch Size: Batch size controls the amount of data processed at one time. In batch processing mode, data is grouped according to a certain batch size and then processed in batches. A larger batch size can reduce the number of data reads and writes, improving data processing efficiency, but it will also increase the load on memory and network. A smaller batch size can reduce processing latency, but may lead to frequent data read and write operations.

[0053] When using CDC technology for distributed data synchronization, specific synchronization parameters such as shard size, parallelism, and batch size are usually manually configured, and the specific parameter values ​​can only be set based on experience.

[0054] Since the source and target ends of different jobs are not the same, many job-related indicators and parameters, such as the total amount of data, the size of a single data item, and the resource utilization of each node, are different. If parameters such as shard size, parallelism, and batch size are set manually based on experience, it is impossible to achieve dynamic adaptation of these parameters to the actual business scenario. In some cases, unreasonable parameter settings may even lead to job failure.

[0055] To address the aforementioned technical problems, the present invention aims to provide a dynamic adjustment scheme for synchronization parameters. The basic idea of ​​the invention is to collect historical CDC synchronization job information, such as endpoint information, synchronization parameters, node resource information, and task performance indicators, as training sample data. This data is used to train a machine learning prediction model for batch size. When scheduling CDC synchronization jobs, the trained prediction model is used to predict the optimal batch size parameter based on the job information within the specified time interval. The batch size parameter of the CDC synchronization job is then dynamically set, making the batch size synchronization parameter more suitable for actual business scenarios and improving the intelligence of synchronization configuration and the efficiency of synchronization.

[0056] Based on the basic idea of ​​the present invention, the present invention provides the following embodiments. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0057] Figure 1 This is a flowchart illustrating the steps of a method for dynamically adjusting synchronization parameters according to an embodiment of the present invention.

[0058] Step 101. Monitor data changes submitted to the distributed stream processing framework cluster. Capture CDC synchronization job tasks and collect job information from historical CDC synchronization jobs to build a training sample set. Job information includes: end point information, synchronization parameters, synchronization node resource information, and task performance indicators.

[0059] This invention constructs a training sample set for training a regression prediction model by monitoring and collecting historical CDC synchronous task information. The regression prediction model is a supervised machine learning model, and can be a linear regression model, multinomial regression model, ridge regression model, decision tree regression model, support vector regression (SVR) model, Lasso regression model, etc.

[0060] The job information for CDC synchronization tasks includes: information on both endpoints, synchronization parameters, synchronization node resource information, and task performance metrics, among which:

[0061] Endpoint information: refers to the relevant information of the data source and data destination of the CDC synchronization task, including but not limited to: source and destination database types, host IP addresses of the databases at both ends, data source table and data destination table;

[0062] Synchronization parameters refer to the synchronization parameters configured for CDC synchronization jobs, including but not limited to: fragment size, parallelism, and batch size;

[0063] Synchronization node resource information refers to the resource usage information of the nodes that execute CDC synchronization tasks, including but not limited to: the usage of resources such as CPU, memory, and network bandwidth of the executing nodes;

[0064] Task performance metrics refer to the performance indicators that reflect the performance of nodes executing CDC synchronization tasks, including but not limited to: data processing latency, data throughput, the degree of data distribution balance for each parallel task, and the average size of each data item.

[0065] Data distribution balance measures whether the amount of data processed across different execution nodes is balanced. If the data is unevenly distributed across the execution nodes, some execution nodes may be overloaded while others are relatively idle, which can negatively impact the overall system performance.

[0066] Step 102. Train the regression prediction model using the training sample set. The input feature values ​​of the regression prediction model are constructed from job information excluding batch size, and the output is the label value of the batch size.

[0067] The feature values ​​of the constructed training sample set include the other information in the aforementioned job information excluding the batch size. Since it is supervised learning, the batch size is used to label the samples. Through training the model, a mapping relationship between the feature values ​​constructed from the job information and the batch size is established.

[0068] Step 103. Before scheduling the CDC synchronization job to be scheduled, calculate the expected data throughput and expected data processing latency based on the historical job information of the CDC synchronization job to be scheduled within the specified time range before the current time.

[0069] Step 104. Using the job information of the CDC synchronization job to be scheduled, excluding data throughput, data processing latency, and batch size, as well as the expected data throughput and expected data processing latency, construct feature values, input the feature values ​​into the regression prediction model, use the batch size output by the regression prediction model as the batch size of the CDC synchronization job to be scheduled, and schedule the CDC synchronization job to be scheduled.

[0070] During initial scheduling, the synchronization parameters of the CDC synchronization jobs to be scheduled can use the system's default configuration values. After the cluster has been executing CDC synchronization jobs for a period of time, due to the different resource information of each execution node, the performance indicators of the synchronization tasks executed by each execution node will also be different. Therefore, this invention uses a trained regression prediction model to predict the batch size of the CDC synchronization jobs to be scheduled based on the actual business environment of each node. This enables dynamic adaptation to the available resource situation of each node, thereby improving the intelligence and synchronization efficiency of CDC synchronization job scheduling.

[0071] Step 105. The training sample set is continuously enriched by collecting historical CDC synchronous task information, and the regression prediction model is optimized through incremental training.

[0072] [Model Training Example]

[0073] In one embodiment of the present invention, a linear regression model is selected as the aforementioned regression prediction model. Linear regression is a supervised learning algorithm used to establish the relationship between a continuous dependent variable and one or more independent variables. The goal of the model is to find a best-fitting straight line or a best-fitting hyperplane to minimize the error between the predicted and observed values. In this embodiment, the dependent variable is the configuration parameter or performance index in the job information excluding the batch size, and the independent variable is the batch size.

[0074] The training of the model consists of the following steps:

[0075] Step S11. Collect job information of historical CDC synchronization tasks, construct a training sample set, and use the batch size of each parallel CDC synchronization task as the label of the training sample (equivalent to labeling the sample data).

[0076] Step S12. Train the linear regression model using the training sample set;

[0077] For a simple linear regression model, the model is equivalent to a linear equation, usually expressed as:

[0078] y = β0 + β1x1 + β2x2 + ... + β i x i +…+β n x n

[0079] Where y is the batch size, x i Let β be the i-th information item in the job information, β0 be the intercept, and n be the number of information items in the job information excluding batch size. i It is the slope of the i-th information item.

[0080] The goal of training the model is to find the optimal set of (β0, β1, ..., β) i ..., β n Model parameters are set to make the model's predictions fit the observed data as closely as possible. This is usually done using the least squares method, which minimizes the sum of squared residuals between the observed values ​​and the model's predictions, and can be calculated using mathematical or computational tools.

[0081] Step S13. Evaluate the predictive performance of the model using an evaluation algorithm to determine the optimal model parameters;

[0082] Once the model is fitted, its performance needs to be evaluated. This is usually done by calculating metrics such as the sum of squared residuals, mean squared error (MSE), and root mean squared error (RMSE). The model parameters of the regression prediction model are then determined based on the evaluation results.

[0083] After the model parameters are finalized, the trained regression prediction model can be applied to the actual business system. The model input feature values ​​are extracted from the job information of each CDC synchronous job to be scheduled, thereby predicting the optimal batch size of the CDC synchronous job to be scheduled, which is the synchronization parameter.

[0084] [Example of a method for determining expected data throughput and expected data processing latency]

[0085] Since each parallel task in a CDC synchronization job is executed on different nodes in a distributed stream processing framework cluster, the resource conditions and performance metrics of each execution node are not entirely the same. If the execution node is used as a parameter of the model, because it is not numerical data, it cannot effectively distinguish the impact of different nodes on the model, leading to a decrease in model accuracy. Therefore, in one embodiment of the present invention, the collected task performance metrics are classified according to the execution node of each parallel task, and the classified data is used to train a regression prediction model for each node separately. For example, if there are three nodes A, B, and C in the cluster, the job information of each parallel task (including endpoint information, synchronization parameters, synchronization node resource information, and task performance metrics, etc.) is classified according to the execution node. The job information of execution node A is used to train regression prediction model A for node A, the job information of execution node B is used to train regression prediction model B for node B, and the job information of execution node C is used to train regression prediction model C for node C, thus obtaining three models: Model A, Model B, and Model C. When using the models, the specific model to be used for prediction is selected based on the scheduling results.

[0086] Since the purpose of this invention is to reduce data processing latency and increase data throughput by predicting the optimal batch size, when using a regression prediction model to predict the batch size of a scheduled CDC synchronous job, it is necessary to predetermine the expected data throughput and expected data processing latency. When constructing the input feature values ​​of the model based on the job information, it is necessary to use information other than data throughput, data processing latency, and batch size from the job information of the scheduled CDC synchronous job task, combined with the expected data throughput and expected data processing latency, to construct the input feature values ​​of the model. Then, the constructed input feature values ​​are input into the regression prediction model so that the regression prediction model outputs the optimal batch size parameter for the current scheduled CDC synchronous job task.

[0087] Figure 2 This is a flowchart illustrating the steps of a method for calculating expected data throughput and expected data processing latency in one embodiment of the present invention, including:

[0088] Step S21. For each execution node in the cluster, filter out the job information records corresponding to the N smallest data processing delays of the CDC synchronization job on that execution node within a specified time range before the current time; where N is a positive integer greater than 1.

[0089] A job information record is a historical record of a CDC synchronization job task scheduled to be executed on an execution node.

[0090] Step S22. For each execution node in the cluster, filter out the job information records corresponding to the top M largest data throughputs of the CDC synchronization job on that execution node within a specified time range before the current time; where M is a positive integer greater than 1, and N can be equal to M, for example, both N and M are 10.

[0091] Among the top-ranked job information records extracted for data processing latency and data throughput, there may be identical job information records or completely different records. The simultaneous appearance of these records in both performance metrics indicates high data synchronization efficiency for the task. To increase the weight of these job information records, overlapping records in the two rankings can be retained. Thus, each execution node will select a total of N+M job information records for calculating expected data processing latency and expected data throughput.

[0092] Step S23. For each execution node in the cluster, calculate the average data processing latency and data throughput in the filtered job information records, and remove outliers based on the standard deviation;

[0093] This step first calculates the average data processing latency and data throughput of the selected N+M job information records. Then, it sequentially checks whether the deviation of each record's data processing latency and data throughput from the average exceeds a specified multiple (e.g., twice) of the standard deviation. If it does, the corresponding record is removed as an outlier. The specific deviation exceeding several times the standard deviation is considered an outlier and can be determined based on experiments or historical experience combined with the actual situation.

[0094] Step S24. For each execution node in the cluster, the average of the data processing latency and data throughput of the remaining job information records after removing outliers is determined as the expected data processing latency and expected data throughput of that execution node.

[0095] After obtaining the expected data processing latency and expected data throughput, the input feature values ​​of the regression prediction model are constructed by combining the remaining information items in the job information of the CDC synchronous job to be scheduled, excluding data processing latency, data throughput and batch size. These feature values ​​are then input into the model to obtain the optimal batch size of the task to be scheduled, and assigned to the CDC synchronous job to be scheduled.

[0096] Figure 3 This is a schematic diagram of an electronic device 300 for implementing the dynamic adjustment method for synchronization parameters provided by the present invention, according to an embodiment of the present invention. The device includes a processor 310, such as a central processing unit (CPU), a communication bus 320, a communication interface 340, and a memory 330. The processor 310 and the memory 330 can communicate with each other via the communication bus 320. The memory 330 stores a computer program, which, when executed by the processor 310, can implement one or more steps in the dynamic adjustment method for synchronization parameters provided by the present invention.

[0097] Memory refers to a device based on a storage medium used to store computer programs and / or data. It can be volatile memory (VM, often called main memory) or non-volatile memory (NVM). Main memory refers to internal storage that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as a storage medium for temporary data of the operating system and other running programs. Main memory can be Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), etc. Non-volatile memory refers to memory that uses persistent storage media, characterized by large capacity and persistent data retention. It can be Storage Class Memory (SCM), Solid State Drive (SSD), NAND flash memory, hard disks, etc. SCM is an industry term for a new type of storage medium that falls between main memory and flash memory. It is a composite storage technology that combines the characteristics of persistent storage and main memory, with access speeds slower than DRAM but faster than SSDs.

[0098] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory (or non-persistent) memory. The method can be implemented using standard programming techniques, including a non-transitory storage medium configured with a computer program within the computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit. Moreover, the operations of the processes described in this invention can be performed in any suitable order unless otherwise indicated by the invention or otherwise clearly contradicted by the context. The processes (or variations and / or combinations thereof) described in this invention can be executed under the control of one or more computer systems configured with executable instructions and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0100] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention also includes the computer itself.

[0101] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for dynamically adjusting synchronization parameters, characterized in that, The method includes: The monitoring and submission of data changes to the distributed stream processing framework cluster captures CDC synchronization tasks. It collects historical CDC synchronization task information to construct a training sample set. The task information includes: endpoint information, synchronization parameters, synchronization node resource information, and task performance metrics. The endpoint information refers to the relevant information of the data source and data destination of the CDC synchronization task. The synchronization parameters refer to the synchronization parameters configured for the CDC synchronization task. The synchronization node resource information refers to the resource usage information of the node executing the CDC synchronization task. The task performance metrics refer to the performance indicators that reflect the performance of the node executing the CDC synchronization task. The regression prediction model is trained using a training sample set. The input feature values ​​of the regression prediction model are constructed from job information excluding batch size, and the output is the label value of the batch size. Before scheduling the CDC synchronization job to be scheduled, calculate the expected data throughput and expected data processing latency based on the historical job information of the CDC synchronization job to be scheduled within a specified time range before the current time. Using the job information of the CDC synchronization job to be scheduled, excluding data throughput, data processing latency, and batch size, other job information, as well as expected data throughput and expected data processing latency, are used to construct feature values. The feature values ​​are then input into the regression prediction model, and the batch size output by the regression prediction model is used as the batch size of the CDC synchronization job to be scheduled. The CDC synchronization job to be scheduled is then executed.

2. The method according to claim 1, characterized in that, The endpoint information includes: source and destination database types, host IP addresses of the databases at both ends, data source table, and data destination table; The synchronization parameters include: fragment size, parallelism, and batch size; The synchronization node resource information includes: the usage of CPU, memory, and network bandwidth resources of the execution node; The task performance metrics include: data processing latency, data throughput, data distribution balance, and average size of each data item.

3. The method according to claim 1, characterized in that, For different execution nodes in the distributed stream processing framework cluster, the job information of the historical CDC synchronous job tasks of each execution node is collected, and the training sample set of each execution node is constructed respectively. The regression prediction model for each execution node is trained using the training sample set of each execution node. When scheduling CDC synchronization jobs, the expected data throughput and expected data processing latency are calculated for each execution node in the cluster, and feature values ​​are constructed. Each execution node uses its own regression prediction model to predict the batch size of each execution node, and schedules the scheduled CDC synchronization job task based on the predicted batch size.

4. The method according to claim 3, characterized in that, The method for calculating the expected data throughput and expected data processing latency for each execution node in the cluster is as follows: Filter out the N smallest data processing delay records for the CDC synchronization job on the execution node within a specified time range before the current time; where N is a positive integer greater than 1. Filter out the job information records corresponding to the top M largest data throughputs of the CDC synchronization job on the execution node within a specified time range before the current time; where M is a positive integer greater than 1; Calculate the average data processing latency and data throughput of the selected job information records, and remove job information records corresponding to outliers based on the standard deviation; For each execution node in the cluster, the average of the data processing latency and data throughput of the remaining job information records after removing outliers is determined as the expected data processing latency and expected data throughput of that execution node.

5. The method according to claim 1, characterized in that, The method further includes: The training sample set is enriched by continuously collecting historical CDC synchronous task information, and the regression prediction model is optimized through incremental training.

6. The method according to claim 1, characterized in that, The regression prediction model is a linear regression model, a multinomial regression model, a ridge regression model, a decision tree regression model, a support vector regression (SVR) model, or a Lasso regression model.

7. A device for dynamically adjusting synchronization parameters, characterized in that, The device includes: The sample set construction module is used to monitor and capture data changes submitted to the distributed stream processing framework cluster for CDC synchronization job tasks. It collects job information of historical CDC synchronization jobs to construct a training sample set. The job information includes: end endpoint information, synchronization parameters, synchronization node resource information, and task performance indicators. The end endpoint information refers to the relevant information of the data source end and data destination end of the CDC synchronization job task. The synchronization parameters refer to the synchronization parameters configured for the CDC synchronization job task. The synchronization node resource information refers to the resource usage information of the node executing the CDC synchronization job task. The task performance indicators refer to the performance indicators that can reflect the performance of the node executing the CDC synchronization job task. The model training module is used to train a regression prediction model using a training sample set. The input feature values ​​of the regression prediction model are constructed from job information excluding batch size, and the output is the label value of the batch size. The expected metrics calculation module is used to calculate the expected data throughput and expected data processing latency of the CDC synchronization job to be scheduled before scheduling the CDC synchronization job to be scheduled, based on the historical job information of the CDC synchronization job to be scheduled within a specified time range before the current time. The scheduling module is used to construct feature values ​​from the job information of the CDC synchronization job to be scheduled, excluding data throughput, data processing latency, and batch size, as well as the expected data throughput and expected data processing latency. The feature values ​​are input into the regression prediction model, and the batch size output by the regression prediction model is used as the batch size of the CDC synchronization job to be scheduled. The CDC synchronization job to be scheduled is then executed.

8. The apparatus according to claim 7, characterized in that, The sample set construction module collects the job information of the historical CDC synchronous job tasks of different execution nodes in the distributed stream processing framework cluster, and constructs the training sample set for each execution node. The model training module uses the training sample set of each execution node to train the regression prediction model of each execution node; When scheduling CDC synchronization tasks, the expected indicator calculation module calculates the expected data throughput and expected data processing latency for each execution node in the cluster and constructs feature values. The scheduling module uses the respective regression prediction model of each execution node to predict the batch size of each execution node, and schedules the scheduled CDC synchronization job task to be executed based on the predicted batch size.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a storage medium, and a communication bus, wherein the processor, the communication interface, and the storage medium communicate with each other through the communication bus; Storage medium used to store computer programs; A processor, when executing a computer program stored on a storage medium, implements the method of any one of claims 1-6.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1 to 6.

Images