Scheduling method and device based on migration rate feedback, storage medium and program product

Abstract

Embodiments of the present application provide a scheduling method and device based on migration rate feedback, a storage medium and a program product, relating to the technical field of data migration. The method comprises: in the data migration process, asynchronously collecting a throughput index of a migration task and a resource load index of a target database; applying a sliding window weighted average algorithm to the collected indexes respectively to generate a smoothed throughput index and a smoothed resource load index; generating a multi-index adjustment coefficient according to the deviation between the smoothed throughput index and a preset target throughput, and the smoothed resource load index; multiplying the execution intensity parameter of the migration task by the multi-index adjustment coefficient according to the multi-index adjustment coefficient, and constraining the adjusted execution intensity parameter within a preset safe threshold range. The method of the present application realizes the adaptive adjustment of the migration parameter, and improves the migration throughput on the premise of ensuring the smoothness of the target database load.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a scheduling method, device, storage medium and program product based on migration rate feedback. Background Technology

[0002] As enterprises deepen their digital transformation, large-scale database upgrades, cross-cloud migrations, and system integrations have become commonplace, requiring the migration of massive amounts of existing data from traditional databases to new databases or the cloud. These scenarios face long migration cycles and real-time fluctuations in resources such as network bandwidth and target database load. Ensuring efficient migration while maintaining business continuity has become a core challenge.

[0003] Current mainstream data migration solutions primarily employ static rate limiting or fixed priority scheduling strategies. Specifically, parameters such as the number of concurrent threads and the upper limit of data transfer rate for migration tasks are typically preset, and multi-database and multi-table migration tasks are executed sequentially according to a manually defined priority order. Some solutions introduce breakpoint resumption and timed retry mechanisms to handle connection interruptions or write failures during the migration process.

[0004] However, the above solutions are insufficient to improve global migration efficiency while ensuring the stability of the target library. Summary of the Invention

[0005] This application provides a scheduling method, device, storage medium, and program product based on migration rate feedback to solve the technical problem of improving global migration efficiency while ensuring the stability of the target library.

[0006] In a first aspect, embodiments of this application provide a scheduling method based on migration rate feedback, the method comprising:

[0007] During the data migration process, asynchronously collect the throughput metrics of the migration task and the resource load metrics of the target database;

[0008] The sliding window weighted average algorithm is applied to the collected throughput and resource load metrics respectively to generate smoothed throughput and resource load metrics. In the sliding window weighted average algorithm, data points with more recent sampling times are given higher weights.

[0009] Based on the deviation between the smoothed throughput index and the preset target throughput, as well as the smoothed resource load index, a multi-index adjustment coefficient that integrates throughput feedback and resource load status is generated.

[0010] Based on the multi-index adjustment coefficient, the execution intensity parameter of the migration task is multiplied by the multi-index adjustment coefficient, and the adjusted execution intensity parameter is constrained within the preset safety threshold range.

[0011] In this embodiment, by combining asynchronous data acquisition with sliding window weighted averaging, the interference of instantaneous fluctuations in throughput and resource load metrics is eliminated, enabling scheduling decisions to accurately reflect the true changing trends of the system. By fusing the smoothed throughput deviation with resource load status to generate multi-metric adjustment coefficients, a joint optimization objective of migration efficiency and target database stability is constructed, avoiding target database overload or resource idleness caused by adjusting a single metric. Through the coordinated control of multiplicative adjustment and boundary constraints, while ensuring smooth parameter convergence, the execution intensity is limited to a safe threshold range, preventing system fluctuations caused by uncontrolled adjustment. Thus, the execution intensity of the migration task is adaptively adjusted to resource fluctuations and performance deviations, improving throughput efficiency in long-cycle, large-scale data migration scenarios while ensuring the business continuity of the target database.

[0012] In one possible implementation, a multi-indicator adjustment coefficient that integrates throughput feedback and resource load status is generated, including:

[0013] The deviation between the smoothed throughput metric and the preset target throughput is calculated and used as the throughput error.

[0014] Apply proportional-integral-derivative control to the throughput error to generate the basic adjustment quantity;

[0015] Obtain the preset load weight coefficient corresponding to the resource load index;

[0016] Based on the basic adjustment amount, load weight coefficient, and smoothed resource load index, a multi-index adjustment coefficient is generated by weighting.

[0017] In this implementation, this embodiment incorporates migration rate deviation and target database load status into the PID control model, generating a multi-indicator adjustment coefficient that integrates throughput feedback and resource load status. This coefficient reflects both the deviation of migration efficiency from the expected target and incorporates protection for the stability of the target database, providing a decision-making basis that balances efficiency and stability for subsequent adjustment of execution intensity parameters.

[0018] In one possible implementation, the execution intensity parameters include the number of concurrent threads and / or the batch write volume;

[0019] In addition, the weighted generation of multi-indicator adjustment coefficients also includes:

[0020] Obtain the priority level of the migration task;

[0021] Priority weight coefficients are generated based on priority levels, and these priority weight coefficients are positively correlated with priority levels.

[0022] The priority weighting coefficient is weighted together with the basic adjustment amount, the load weighting coefficient, and the smoothed resource load index to generate a multi-index adjustment coefficient.

[0023] In this implementation, the task priority is embedded into a multi-indicator fusion scheduling decision model. This enables the scheduling decision to not only perceive system load and migration efficiency but also business importance, achieving three-dimensional joint optimization of efficiency, stability, and priority. This improves the migration efficiency of core businesses and the rational utilization of system resources in complex multi-task migration scenarios.

[0024] In one possible implementation, the method further includes:

[0025] Monitor the resource load metrics of the target database and / or the write latency of the migration task;

[0026] When the resource load indicator exceeds the preset load threshold, or the write latency exceeds the preset latency threshold, or the smoothed throughput indicator decreases at a rate exceeding the preset decrease threshold, a rollback adjustment is triggered.

[0027] The rollback adjustment includes at least one of the following: restoring the execution intensity parameter to the value of the previous adjustment cycle, reducing the execution intensity parameter according to the preset rollback algorithm, and suspending the execution of migration tasks with a priority level lower than the preset level threshold.

[0028] In this implementation, the anomaly rollback mechanism provided in this embodiment automatically triggers degradation protection when the target database experiences overload, a surge in write latency, or a continuous decline in migration throughput: it rolls back to stable parameters for mild overload, smoothly reduces speed using a backoff algorithm for sustained overload, and suspends low-priority tasks when resource contention is severe. This mechanism enables the migration task to have self-healing capabilities, quickly restoring a stable operating state without manual intervention and ensuring business continuity for the target database.

[0029] In one possible implementation, the method further includes:

[0030] In response to a migration task having a priority level higher than the preset high priority threshold, the adjustment range of the execution intensity parameter is increased;

[0031] In response to a migration task's priority level falling below a preset low-priority threshold, the adjustment range of the execution intensity parameter is reduced, and / or the migration task is paused.

[0032] In this implementation, the priority-differentiated fallback mechanism provided in this embodiment allows the system to dynamically allocate scarce resources based on task importance during abnormal degradation. High-priority tasks gain stronger anti-interference capabilities, while low-priority tasks bear more degradation costs, thereby achieving differentiated protection of the migration service level agreement as a whole and improving resource utilization efficiency in multi-task concurrent migration scenarios.

[0033] In one possible implementation, the sampling period used for asynchronous acquisition is adjusted according to the degree of fluctuation in the throughput metric;

[0034] The degree of volatility was calculated by applying a sliding window variance algorithm to the throughput metric.

[0035] Furthermore, the sampling period is inversely proportional to the degree of fluctuation.

[0036] In this implementation, the adaptive sampling period adjustment mechanism provided in this embodiment enables the system to maintain high sensitivity when the migration rate fluctuates drastically, promptly capturing abnormal states for scheduling decisions; and to proactively reduce the sampling frequency when the migration rate is stable, minimizing unnecessary metric collection overhead. This mechanism reduces the resource collector's consumption of CPU, memory, I / O, and other system resources without affecting scheduling real-time performance, thereby improving the overall scalability and resource utilization efficiency of the migration system.

[0037] In one possible implementation, the method further includes:

[0038] Obtain the feature type of the migration task, which includes at least one of cold data, hot data, and mixed data;

[0039] Configure at least one of the proportional coefficient, integral coefficient, and derivative coefficient of proportional-integral-derivative control according to the feature type;

[0040] Among them, the proportion coefficient of cold data migration tasks is higher than that of hot data migration tasks, and the integral coefficient of cold data migration tasks is lower than that of hot data migration tasks.

[0041] In this implementation, by employing an adaptive configuration strategy for PID parameters bound to task characteristic types, this embodiment enables the same PID control model to maintain good control performance in significantly different data migration scenarios, such as cold data and hot data. Cold data migration achieves faster start-up speed and backlog suppression capabilities, while hot data migration achieves a smoother operating trajectory and higher steady-state accuracy. This mechanism eliminates the need for manual parameter tuning, enhancing the PID controller's adaptability to various data migration tasks.

[0042] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0043] The memory stores instructions that the computer executes;

[0044] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0045] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0046] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 A schematic diagram of the architecture of a scheduling system based on migration rate feedback provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating a scheduling method based on migration rate feedback provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram illustrating the process of generating multi-index adjustment coefficients, as provided in an embodiment of this application.

[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] It should be noted that the scheduling method, device, storage medium and program product based on migration rate feedback provided in this application can be used in the field of data migration technology, or in any field other than data migration technology. This application does not limit the application field of the scheduling method, device, storage medium and program product based on migration rate feedback.

[0054] The specific application scenarios of this application are applicable to large-scale existing data migration projects with high requirements for migration efficiency, system stability, and multi-task collaboration, including the following typical scenarios:

[0055] During the upgrade and replacement of core business systems from traditional commercial databases to domestically produced databases, tens of terabytes of transaction history data and existing business data need to be migrated from the source database to the target database. The migration window often overlaps with peak daytime business hours. This application is applicable to such long-term, large-scale migration tasks that require maintaining stable performance of the target database under fluctuating business loads.

[0056] In cross-cloud platform data migration scenarios, enterprises migrate massive amounts of data from local data centers or private clouds to public cloud platforms. Due to the unstable bandwidth of public network transmission links and the rate limiting mechanism of cloud write interfaces, this application can adapt to such complex environments with drastic fluctuations in network conditions, ensuring the continuous progress of migration operations.

[0057] In hybrid cloud architecture integration or multi-heterogeneous system merging scenarios, it is necessary to aggregate source data from multiple business units to a unified target data platform. Different source databases have different hardware configurations and load characteristics, and each migration task has different priorities in terms of business importance. This application is applicable to complex scheduling scenarios involving multiple concurrent tasks, significant resource contention, and the need to prioritize and protect high-priority tasks.

[0058] In regulatory compliance transformation projects in industries such as finance and telecommunications, data migration must be completed without interrupting external services, and the target database must continuously support front-end transaction queries and writes. This application applies to sensitive production environments with high requirements for the stability of the target database and where business fluctuations due to migration load are not permitted.

[0059] For the above application scenarios, existing technical solutions typically adopt the following implementation methods:

[0060] One possible implementation is a fixed-parameter migration approach. Before the migration task starts, the system administrator or operations personnel manually set parameters such as the number of concurrent threads, the number of data rows read per batch, and the time interval for writing to the target database for each migration task, based on experience. After the migration task starts, the scheduling module runs continuously according to the set parameter values, for example, a fixed 16 concurrent threads, 5000 rows of data read per batch, and a write operation committed every 2 seconds. Throughout the entire migration cycle, these parameters remain unchanged regardless of whether the network bandwidth is idle or congested, or whether the CPU load of the target database is low or high, until the migration task ends or manual intervention is required.

[0061] Another possible implementation is a pre-assessment recommendation approach. Some commercial data migration platforms perform a brief benchmark test or hardware capability scan before the task officially starts. Specifically, the system extracts a small amount of sample data from the source, tests network transmission rates, and sends probe write requests to the target database. Based on these extracted metrics, it estimates indicators such as the source disk read capability, network link bandwidth limit, and the target database's transactions per second (TPS).

[0062] Subsequently, based on these evaluation results, the system automatically generates a set of recommended configuration parameters such as concurrency, shard size, and rate limiting threshold during the task initialization phase, and uses this static configuration during the migration process. If the resource status changes during the migration, the system will not actively adjust the parameters; the evaluation will only be re-executed when the task is restarted after an abnormal interruption.

[0063] Another possible implementation is the queue backlog feedback adjustment method. This type of scheme is mainly used in streaming data synchronization tools, and its typical architecture sets up a memory buffer queue between the data production module and the data consumption module. The production module is responsible for reading data from the source and putting it into the queue, while the consumption module is responsible for pulling data from the queue and writing it into the target database.

[0064] The system continuously monitors the length of unprocessed data in the queue and sets two watermark thresholds: when the queue backlog exceeds the high watermark threshold, it determines that the consumer's processing capacity is insufficient, and the production module proactively reduces the reading speed or suspends reading; when the queue backlog is below the low watermark threshold, it determines that the consumer is idle, and the production module resumes or increases the reading speed. The adjustment actions are typically simple speed switching, such as "full speed, half speed, pause," and the adjustment is based solely on the queue length metric.

[0065] However, the solution described above has the following technical problems:

[0066] The scheduling strategies for migration tasks are all statically preset or coarsely based on a single metric, unable to dynamically and adaptively adjust according to real-time changes in network bandwidth, target database load, and source data characteristics. Fixed-concurrency solutions, during long migration cycles, suffer from underutilization of resources when idle and potential write blocking to the target database during resource shortages. Static recommendation solutions rely excessively on brief pre-migration assessments, while actual operating environments differ significantly from benchmark conditions, and the pre-recommended values ​​may become ineffective during peak business periods.

[0067] Furthermore, existing solutions lack proactive protection mechanisms for the stability of the target library. Neither fixed concurrency nor static recommendation incorporates key metrics such as the target library's real-time TPS, CPU utilization, and IO wait times into the scheduling decision. This leads to frequent issues such as lock contention and write timeouts caused by migration overload, while queue backlog solutions only passively slow down after overload occurs, exhibiting a severe lag effect.

[0068] In multi-task concurrent scenarios, existing technologies also have scheduling blind spots. Different migration tasks can only be executed serially in a manually defined fixed order or with static priorities, lacking any guarantee of preferential treatment for high-priority tasks. When migration tasks of different priorities run concurrently, resource contention prevents high-priority tasks from obtaining migration throughput commensurate with their own business importance.

[0069] Furthermore, existing solutions are insufficient in handling abnormal states. When the target database load suddenly increases, network quality deteriorates, or migration rate abnormally decreases, the system cannot automatically identify and execute degradation and rollback strategies, often directly leading to migration task failure or data duplication, requiring manual intervention for recovery.

[0070] The scheduling method based on migration rate feedback provided in this application aims to solve the aforementioned technical problems of existing technologies. It constructs an adaptive scheduling mechanism with real-time migration rate and resource status feedback as its core. This method abandons the traditional static preset or single-index coarse scheduling mode, and instead uses the real-time running status during the migration process as the basis for scheduling decisions.

[0071] Specifically, this solution constructs a real-time feedback loop reflecting the actual operating status of the system by asynchronously collecting multi-dimensional indicators such as migration rate, target library load, and network bandwidth. Based on this, a multi-indicator fusion control model is introduced, using migration rate and resource status deviation as inputs to calculate the optimal adjustment amounts for concurrency and batch size, ensuring that migration behavior always adaptively adjusts to resource fluctuations.

[0072] To balance target library stability and global migration efficiency, this solution incorporates target library load metrics into the control objectives. While pursuing migration throughput, it constrains the resource impact on the target library, avoiding lock contention and write blocking caused by overload. Furthermore, in multi-task concurrent scenarios, task priority is embedded as a scheduling weight in the control model, ensuring that high-priority tasks receive preferential treatment in resource contention.

[0073] Furthermore, this solution incorporates anomaly detection and rollback mechanisms. When a sudden surge in target library load or an abnormal drop in migration rate is detected, a degradation strategy is automatically implemented and a smooth recovery is achieved, preventing migration task interruption or data duplication. Through this closed-loop feedback and adaptive adjustment, automated scheduling throughout the entire migration cycle is realized.

[0074] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0075] First, combine Figure 1 This application introduces a scheduling system based on migration rate feedback provided in its embodiments. Figure 1 This is a schematic diagram of the architecture of a scheduling system based on migration rate feedback, provided as an embodiment of this application. Figure 1 As shown, the scheduling system in this embodiment includes:

[0076] The resource collector is used to asynchronously collect throughput metrics of the migration task and resource load metrics of the target database during the data migration process.

[0077] The rate monitor is used to apply a sliding window weighted average algorithm to the collected throughput and resource load metrics respectively, generating smoothed throughput and resource load metrics.

[0078] An adaptive scheduler is used to generate a multi-index adjustment coefficient that integrates throughput feedback and resource load status based on the deviation between the smoothed throughput index and the preset target throughput, as well as the smoothed resource load index. Based on the multi-index adjustment coefficient, the execution intensity parameter of the migration task is multiplied by the multi-index adjustment coefficient, and the adjusted execution intensity parameter is constrained within a preset safety threshold range.

[0079] The priority manager provides priority levels for migration tasks and generates priority weight coefficients that are positively correlated with the priority levels.

[0080] The read executor and write executor are used to read data from the source and write data to the target database, respectively, and their corresponding execution intensity parameters are dynamically controlled by the adaptive scheduler.

[0081] This embodiment achieves a dynamic balance between migration efficiency and target library stability through the collaborative work of the above components, forming a closed-loop adaptive scheduling architecture of "acquisition-smoothing-feedback-adjustment-constraint".

[0082] The following is combined Figure 2 This application introduces a scheduling method based on migration rate feedback provided in its embodiments. Figure 2 This is a flowchart illustrating a scheduling method based on migration rate feedback, provided as an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0083] S201. During the data migration process, asynchronously collect the throughput metrics of the migration task and the resource load metrics of the target database.

[0084] Specifically, in the process of migrating massive amounts of existing data from the source database to the target database, this embodiment uses a resource collector to asynchronously execute collection tasks. The collection tasks are decoupled from the main migration process and run as independent threads or processes, avoiding blocking data read and write operations.

[0085] Throughput metrics are used to characterize the execution efficiency of migration tasks, including but not limited to rows migrated per second (Rows / s) or data volume migrated per second (MB / s). Resource load metrics are used to characterize the real-time load status of the target database, including but not limited to CPU load and I / O load.

[0086] This embodiment does not impose a specific limitation on the sampling period, which can be flexibly configured according to application requirements. For example, it can be set to a fixed period of 3 seconds, or an adaptive period adjustment strategy can be adopted.

[0087] S202. Apply the sliding window weighted average algorithm to the collected throughput and resource load indicators respectively to generate smoothed throughput and smoothed resource load indicators.

[0088] Since the raw metrics collected often fluctuate instantaneously due to factors such as network jitter and momentary disk busy, directly using them for scheduling decisions may lead to parameter instability. Therefore, this embodiment introduces a sliding window weighted average algorithm to smooth the raw metrics.

[0089] Specifically, a fixed-length time window is maintained for each indicator, containing the most recent N sampled values. When calculating the smoothed value at the current moment, a weight is assigned to each sample point within the window, with data points more recently sampled receiving a higher weight. For example, an exponentially weighted moving average algorithm can be used, where the weight coefficients decrease exponentially with the time elapsed since the sampling point.

[0090] Through the above smoothing process, the smoothed throughput and smoothed resource load indicators generated in this embodiment can truly reflect the changing trend of the system and effectively filter out the interference of instantaneous outliers.

[0091] S203. Based on the deviation between the smoothed throughput index and the preset target throughput, as well as the smoothed resource load index, generate a multi-index adjustment coefficient that integrates throughput feedback and resource load status.

[0092] After obtaining the smoothed throughput and resource load metrics, the adaptive scheduler in this embodiment initiates the adjustment coefficient generation process.

[0093] First, the relative deviation between the smoothed throughput metric and the preset target throughput is calculated as the throughput error. The preset target throughput can be set in advance according to business needs, for example, 50k rows / s.

[0094] Secondly, obtain the preset load weight coefficients corresponding to the resource load indicators. The load weight coefficients are used to quantify the impact of resource load status on the adjustment of execution intensity parameters, and their values ​​can be pre-calibrated based on factors such as the hardware configuration of the target database and business tolerance.

[0095] Finally, the throughput error, load weight coefficient, and smoothed resource load index are input into the multi-index fusion model to generate multi-index adjustment coefficients.

[0096] In one specific implementation, the multi-index fusion model employs a proportional-integral-derivative (PID) control algorithm. First, PID control is applied to the throughput error to generate a basic adjustment quantity. Then, the basic adjustment quantity is superimposed with the resource load index weighted by the load weight coefficient to obtain the final multi-index adjustment coefficient, denoted as K′.

[0097] S204. Based on the multi-index adjustment coefficient, multiply the execution intensity parameter of the migration task by the multi-index adjustment coefficient, and constrain the adjusted execution intensity parameter within the preset safety threshold range.

[0098] After generating the multi-index adjustment coefficient K′, the adaptive scheduler adjusts the execution intensity parameters of the migration task accordingly. The execution intensity parameters include at least one of the number of concurrent threads and the batch write volume; this embodiment uses simultaneous adjustment of both as an example. For ease of description, let the number of concurrent threads used in the current period be denoted as... Batch write volume is Their preset safety threshold ranges are respectively [ , ]and[ , The smoothing factor is λ (0 < λ ≤ 1).

[0099] This embodiment first employs a multiplicative adjustment method to... and Multiply by the multi-index adjustment coefficient K′ respectively to obtain the original adjustment value. = ×K′ and = ×K′. Multiplicative regulation has geometric convergence properties, ensuring that the migration intensity and the regulation coefficient maintain a proportional following relationship.

[0100] To prevent parameter adjustment from going out of control, this embodiment further limits the original adjustment value to a preset safety threshold range using a boundary constraint function. Specifically, for the number of concurrent threads, if... Below Then take higher than Then take If it falls between the two, then take The same applies to batch writes. The number of concurrent threads after boundary constraints is obtained. and batch write volume .

[0101] To suppress frequent parameter oscillations, this embodiment also introduces an exponential smoothing mechanism to perform weighted fusion of the parameters after boundary constraints and the parameters of the previous period: =λ× +(1-λ)× , =λ× +(1-λ)× The closer the smoothing factor λ is to 1, the more sensitive the response; the smaller λ is, the higher the historical weight and the smoother the parameter changes.

[0102] The final result and This refers to the execution intensity parameters issued in this cycle. The read executor and write executor continue to execute the migration task according to the new parameters. The changes in their operating status will be perceived by the next round of indicator collection, thus forming a continuous closed-loop optimization process of "collection-feedback-adjustment-constraint-smoothing".

[0103] The migration rate feedback-based scheduling method provided in this embodiment eliminates the interference of instantaneous fluctuations in throughput and resource load indicators by combining asynchronous data acquisition with sliding window weighted averaging, enabling scheduling decisions to accurately reflect the true changing trend of the system. By fusing the smoothed throughput deviation with the resource load status to generate multi-indicator adjustment coefficients, a joint optimization objective of migration efficiency and target database stability is constructed, avoiding target database overload or resource idleness caused by adjusting a single indicator. Through the coordinated control of multiplicative adjustment and boundary constraints, the execution intensity is limited to a safe threshold range while ensuring smooth parameter convergence, preventing system fluctuations caused by uncontrolled adjustment. Thus, the execution intensity of migration tasks is adaptively adjusted to resource fluctuations and performance deviations, improving throughput efficiency in long-cycle, large-scale data migration scenarios while ensuring the business continuity of the target database.

[0104] Optionally, in one specific embodiment of step S203 above, the multi-index adjustment coefficients are generated through PID control that fuses multiple indices. This part will be described below in conjunction with section 3. Figure 3 This is a schematic diagram illustrating the process of generating multi-index adjustment coefficients, as provided in an embodiment of this application.

[0105] like Figure 3 As shown, this embodiment provides a regulation coefficient generation process based on PID control and multi-index weighted fusion. This process forms a closed-loop control loop of index acquisition, feedback processing, rate control, and regulation feedback.

[0106] The adaptive scheduler obtains the smoothed throughput metric for the current migration task from the rate monitor. And the smoothed resource load metrics of the target database, including at least CPU load. and input / output load .

[0107] The feedback processor calculates the smoothed throughput metric. Compared with the preset target throughput The relative deviation between them is taken as the throughput error e(t):

[0108] =( - ) /

[0109] Preset target throughput The expected migration rate is pre-configured based on business needs, for example, 50k rows / s; This represents the actual migration rate after smoothing for the current period. When e(t) is greater than 0, it indicates that the current migration rate is lower than expected, and the migration intensity needs to be increased; when e(t) is less than 0, it indicates that the current migration rate is higher than expected, and the migration intensity can be appropriately reduced to avoid excessive resource consumption.

[0110] Feedback processor for throughput error The basic adjustment quantity K is generated by applying the PID control algorithm:

[0111] K= × + ×∫e(t)dt+ ×de(t) / dt

[0112] in, This is a proportionality coefficient that determines the strength of the response to the current error; These are the integral coefficients used to eliminate accumulated steady-state errors; These are differential coefficients used to suppress abrupt changes in error trends. By adjusting these three sets of coefficients, the response speed and adjustment characteristics of the feedback loop can be flexibly configured to adapt to the stability requirements of different migration scenarios.

[0113] The feedback processor obtains preset load weight coefficients corresponding to resource load indicators. The load weight coefficients include CPU load weight coefficient β and input / output load weight coefficient γ. The feedback intensity weight coefficient α is used to adjust the degree of influence of the rate feedback path on the final adjustment coefficient.

[0114] Subsequently, the feedback processor multiplies the basic adjustment amount K by the feedback intensity weighting coefficient α, and then superimposes it with the resource load index weighted by the load weighting coefficient to generate a multi-index adjustment coefficient K′:

[0115] K′=α×K+β× +γ×

[0116] The physical meaning of this formula is that when it is necessary to strengthen the impact of migration rate feedback on scheduling decisions, a larger α value can be configured; when it is necessary to strengthen the protection of the target database load, a larger β and γ value can be configured; by adjusting the relative size of the three sets of weight coefficients, the trade-off strategy between efficiency priority and stability priority under different migration scenarios can be flexibly adapted.

[0117] The adaptive rate controller outputs the generated multi-index adjustment coefficient K′ to the data migration system to adjust the execution intensity parameters of the migration task, including the number of concurrent threads and the batch write volume.

[0118] The adjusted execution intensity parameters and their corresponding migration rates and resource load status are fed back to the indicator collection stage, triggering the indicator collection and adjustment coefficient calculation in the next sampling cycle, forming a continuous closed-loop optimization.

[0119] Through the above process, this embodiment integrates migration rate deviation and target database load status into the PID control model, generating a multi-index adjustment coefficient that combines throughput feedback and resource load status. This coefficient reflects both the degree of deviation between migration efficiency and the expected target, and also incorporates protection for the stability of the target database, providing a decision-making basis that balances efficiency and stability for subsequent adjustment of execution intensity parameters.

[0120] Optionally, in one specific embodiment of step S203 above, the weighted generation process of the multi-index adjustment coefficient further incorporates the priority level of the migration task.

[0121] In this embodiment, when the adaptive scheduler executes step S203, it first obtains the priority level of the current migration task. The priority level is used to characterize the importance or urgency of the migration task and can be pre-configured according to business categories. For example, migration tasks corresponding to core business systems are configured as high priority, and non-core systems are configured as low priority.

[0122] The adaptive scheduler generates a corresponding priority weight coefficient based on the priority level. This coefficient is positively correlated with the priority level; that is, tasks with higher priority are assigned a larger priority weight coefficient. This embodiment does not limit the specific value of the priority level or the mapping method between it and the weight coefficient, as long as it ensures that high-priority tasks receive a higher priority weight coefficient.

[0123] After generating the priority weight coefficient, the adaptive scheduler weights it together with the basic adjustment amount, load weight coefficient, and smoothed resource load index generated in the above embodiment to generate the final multi-index adjustment coefficient.

[0124] The physical meaning of this weighted fusion process is that high-priority tasks amplify the contribution of basic adjustments through larger priority weighting coefficients, enabling them to achieve a stronger migration intensity adjustment response under the same throughput error; while low-priority tasks weaken the contribution of basic adjustments through smaller priority weighting coefficients, proactively relinquishing some migration capacity when resources are scarce. At the same time, the target database's load protection mechanism remains independently effective, and regardless of task priority, it must adhere to constraints on the stability of the target database.

[0125] By employing the above method, this embodiment embeds task priority into a multi-indicator fusion scheduling decision model. This enables scheduling decisions to not only perceive system load and migration efficiency but also business importance, achieving three-dimensional joint optimization of efficiency, stability, and priority. This improves the migration efficiency of core businesses and the rational utilization of system resources in complex multi-task migration scenarios.

[0126] Optionally, based on the above scheduling method, this embodiment further introduces an abnormal rollback mechanism to deal with system overload or performance degradation scenarios that may occur during the migration process.

[0127] In this embodiment, during the migration task execution, the resource load indicators of the target database and the write latency of the migration task are continuously monitored. The resource load indicators include at least CPU load and I / O load; the write latency refers to the time taken for a single operation to write data to the target database or the cumulative queuing waiting time.

[0128] Meanwhile, this embodiment also monitors the changing trend of the smoothed throughput metric in real time and calculates its rate of decline per unit time. When any of the following abnormal conditions are detected, the system automatically triggers rollback adjustment: the CPU load or I / O load of the target database exceeds a preset load threshold; the write latency of the migration task exceeds a preset latency threshold; the rate of decline of the smoothed throughput metric exceeds a preset decline threshold.

[0129] The aforementioned thresholds can be preset based on factors such as the target database's hardware configuration, peak and off-peak business hours, and the service level agreement for the migration task. This embodiment does not impose specific limitations on these thresholds.

[0130] After triggering rollback adjustment, the adaptive scheduler selects to execute one or more of the following rollback operations based on the current severity of the anomaly and the system state:

[0131] The first method involves restoring the execution intensity parameters of the current migration task to the values ​​from the previous adjustment cycle. This operation is suitable for scenarios with momentary jitter or mild overload, allowing the system to quickly return to a previously verified stable operating state by canceling the current adjustment action.

[0132] The second approach involves gradually reducing execution intensity parameters according to a pre-defined backoff algorithm. For example, exponential or linear backoff strategies can be used, gradually reducing the number of concurrent threads or batch write volume proportionally at regular intervals until the system load returns to normal or the parameters drop to a safe lower limit. This operation is suitable for scenarios with continuous overload, avoiding impact on the target library through smooth degradation.

[0133] The third approach is to suspend migration tasks with a priority level below a preset threshold. This operation is suitable for scenarios with severe resource contention or where the target database is on the verge of collapse. By proactively relinquishing resource usage by low-priority tasks, it prioritizes ensuring the migration continuity of the core business system and the stability of the target database. Suspended migration tasks can be automatically restarted and resumed by the scheduler once the system load returns to normal.

[0134] After the above rollback operation is completed, the system records the abnormal event and rollback parameter logs for subsequent offline analysis and parameter tuning of the scheduling strategy. Meanwhile, the migration task continues to run, and indicator collection and monitoring continue. The system remains within the closed-loop control framework of "collection-feedback-adjustment-rollback," ready to respond to the next round of normal adjustment or rollback command triggered by another abnormality.

[0135] The anomaly rollback mechanism provided in this embodiment enables the system to automatically trigger degradation protection when the target database experiences overload, a surge in write latency, or a continuous decline in migration throughput: rolling back to stable parameters for mild overload, smoothly reducing speed using a backoff algorithm for sustained overload, and suspending low-priority tasks when resource contention is severe. This mechanism gives the migration task the ability to self-heal from anomalies, quickly restoring a stable operating state without manual intervention and ensuring business continuity for the target database.

[0136] Optionally, based on the above-mentioned abnormal rollback mechanism, this embodiment further implements differentiated rollback strategies for migration tasks with different priority levels.

[0137] In this embodiment, when the adaptive scheduler triggers rollback adjustment, it first obtains the priority level of the current migration task and compares it with the preset high priority threshold and low priority threshold.

[0138] For migration tasks with a priority level higher than the preset high-priority threshold, this embodiment implements a priority enhancement strategy. Specifically, during the rollback adjustment process, the adjustment range of the execution intensity parameters for such tasks is appropriately increased. Specifically, when it is necessary to reduce the migration intensity to alleviate the load on the target library, the reduction range of the concurrent thread count or batch write volume for high-priority tasks is relatively smaller; when the load returns to normal and the migration intensity needs to be restored, the increase range of high-priority tasks is relatively larger. Through this differentiated processing, high-priority tasks can receive more resource allocation in abnormal scenarios, maximizing their migration efficiency.

[0139] For migration tasks with a priority level lower than the preset low priority threshold, this embodiment implements a priority downgrade strategy: on the one hand, the execution intensity parameter of such tasks is reduced by a larger margin according to the preset backoff algorithm, so that system resources can be released more quickly; on the other hand, when resources are severely strained, all migration tasks with a priority level lower than the preset threshold can be suspended directly, and the scheduler will automatically wake them up and resume the migration after the system load returns to normal.

[0140] The aforementioned high-priority and low-priority thresholds can be pre-configured according to business needs. For example, priority levels 1-2 can be set as low priority, and levels 4-5 as high priority. This embodiment does not limit the specific values ​​of the thresholds.

[0141] Through the priority-differentiated fallback mechanism provided in this embodiment, the system can dynamically allocate scarce resources based on task importance during abnormal degradation. High-priority tasks gain stronger anti-interference capabilities, while low-priority tasks bear more degradation costs, thereby achieving differentiated protection of the migration service level agreement as a whole and improving resource utilization efficiency in multi-task concurrent migration scenarios.

[0142] Optionally, based on the above scheduling method, this embodiment further adaptively adjusts the sampling period of asynchronous acquisition to balance real-time performance and system overhead.

[0143] In this embodiment, when the resource collector asynchronously collects throughput metrics, the sampling period is not fixed, but dynamically adjusted according to the fluctuation level of the throughput metrics. The fluctuation level is used to quantify the instability of the migration rate. The greater the fluctuation, the more drastic the change in the migration rate, requiring more frequent collection to detect state changes in a timely manner; the smaller the fluctuation, the more stable the migration rate tends to be, and the collection interval can be appropriately extended to reduce system overhead.

[0144] To quantify the degree of volatility, this embodiment applies a sliding window variance algorithm to the throughput metric. Specifically, a fixed-length time window is maintained for the throughput metric, containing the most recent N sampled values. Each time a new sample is completed, the variance of all sampled values ​​within the window is recalculated, and the result is used as a measure of the volatility in the current period. A larger variance indicates more drastic fluctuations in the throughput metric within the window period; a smaller variance indicates a more stable throughput metric.

[0145] After obtaining the fluctuation level of the current period, the resource collector adjusts the sampling period for the next period accordingly. The sampling period is inversely proportional to the fluctuation level: when the fluctuation level is large, the sampling period is shortened to capture rapid changes in migration rate at a higher sampling frequency; when the fluctuation level is small, the sampling period is lengthened and the sampling frequency is reduced to minimize the consumption of system resources.

[0146] This embodiment does not limit the window length of the sliding window, the specific method of variance calculation, or the inverse mapping relationship between the sampling period and the degree of fluctuation, as long as the sampling period decreases as the degree of fluctuation increases and increases as the degree of fluctuation decreases. For example, multiple sampling period values ​​can be pre-configured, and the level can be switched according to the range of variance; or a continuous function mapping can be used, such as T=T_base / (1+α·Var), where T_base is the base period, Var is the variance, and α is the sensitivity adjustment coefficient.

[0147] Through the adaptive sampling period adjustment mechanism provided in this embodiment, the system can maintain high sensitivity when the migration rate fluctuates drastically, promptly capturing abnormal states for scheduling decisions; when the migration rate is stable, it actively reduces the sampling frequency, reducing unnecessary metric collection overhead. This mechanism reduces the resource collector's consumption of CPU, memory, I / O, and other system resources without affecting the real-time performance of scheduling, thereby improving the overall scalability and resource utilization efficiency of the migration system.

[0148] Optionally, the PID parameter adaptive configuration mechanism provided in this embodiment can be used to set proportional coefficients and integral coefficients differently for different types of data migration tasks, so that the dynamic response characteristics of the PID controller match the data access characteristics of the migration task.

[0149] For cold data migration tasks, typical characteristics include slow source-end read speeds, smooth throughput fluctuations, and insensitivity to transient interference. By configuring a higher proportional coefficient, the controller's instantaneous response to throughput deviations is enhanced, enabling rapid increases in migration intensity when a rate drop is detected, thus avoiding queue backlogs caused by read latency. Simultaneously, configuring a lower integral coefficient prevents excessive accumulation of integral terms due to the inherently low-speed characteristics of cold data migration, avoiding adjustment lag or overshoot caused by integral saturation.

[0150] For hot data migration tasks, typical characteristics include fast source-end read speed, high peak throughput, and sensitivity to system load fluctuations. By configuring a low proportional coefficient, system oscillations caused by over-adjustment due to minor deviations can be avoided; at the same time, a high integral coefficient can be configured to utilize the error accumulation effect of the integral term to eliminate steady-state errors, ensuring that hot data migration can maintain a highly efficient operating state close to the target throughput over a long period of time.

[0151] For mixed data migration tasks, linear interpolation or segmented configuration can be performed between cold and hot parameter configurations based on the proportional distribution of cold and hot data. This embodiment does not impose specific limitations on this.

[0152] Through the aforementioned adaptive configuration strategy of PID parameters bound to task characteristics, this embodiment enables the same PID control model to maintain good control performance in significantly different data migration scenarios, such as cold data and hot data. Cold data migration achieves faster start-up speed and backlog suppression capability, while hot data migration achieves a smoother running trajectory and higher steady-state accuracy. This mechanism requires no manual intervention for parameter tuning, improving the PID controller's adaptability to various data migration tasks.

[0153] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0154] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0155] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0156] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0157] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0161] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0162] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0163] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0164] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0165] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0166] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0167] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A scheduling method based on migration rate feedback, characterized in that, The method includes: During the data migration process, asynchronously collect the throughput metrics of the migration task and the resource load metrics of the target database; The sliding window weighted average algorithm is applied to the collected throughput index and resource load index respectively to generate smoothed throughput index and smoothed resource load index. In the sliding window weighted average algorithm, data points with more recent sampling time are given higher weights. Based on the deviation between the smoothed throughput index and the preset target throughput, and the smoothed resource load index, a multi-index adjustment coefficient that integrates throughput feedback and resource load status is generated. Based on the multi-index adjustment coefficient, the execution intensity parameter of the migration task is multiplied by the multi-index adjustment coefficient, and the adjusted execution intensity parameter is constrained within a preset safety threshold range.

2. The method according to claim 1, characterized in that, The generation of multi-index adjustment coefficients that integrate throughput feedback and resource load status includes: The deviation between the smoothed throughput index and the preset target throughput is calculated as the throughput error. Proportional-integral-derivative control is applied to the throughput error to generate a basic adjustment value; Obtain the preset load weight coefficient corresponding to the resource load index; The multi-index adjustment coefficient is generated by weighting the basic adjustment amount, the load weight coefficient, and the smoothed resource load index.

3. The method according to claim 2, characterized in that, The execution intensity parameters include the number of concurrent threads and / or the batch write volume; Furthermore, the weighted generation of the multi-index adjustment coefficients further includes: Obtain the priority level of the migration task; A priority weight coefficient is generated based on the priority level, and the priority weight coefficient is positively correlated with the priority level; The priority weighting coefficient is weighted together with the basic adjustment amount, the load weighting coefficient, and the smoothed resource load index to generate the multi-index adjustment coefficient.

4. The method according to claim 3, characterized in that, The method further includes: Monitor the resource load metrics of the target database and / or the write latency of the migration task; When the resource load indicator exceeds a preset load threshold, or the write latency exceeds a preset latency threshold, or the smoothed throughput indicator decreases at a rate exceeding a preset decrease threshold, a rollback adjustment is triggered. The rollback adjustment includes at least one of the following: restoring the execution intensity parameter to the value of the previous adjustment cycle, reducing the execution intensity parameter according to a preset rollback algorithm, and suspending the execution of migration tasks whose priority level is lower than a preset level threshold.

5. The method according to claim 4, characterized in that, The method further includes: In response to the migration task having a priority level higher than a preset high priority threshold, the adjustment range of the execution intensity parameter is increased; In response to the migration task having a priority level lower than a preset low priority threshold, the adjustment range of the execution intensity parameter is reduced, and / or the execution of the migration task is suspended.

6. The method according to claim 1, characterized in that, The sampling period used in the asynchronous acquisition is adjusted according to the fluctuation of the throughput indicator; The degree of fluctuation is calculated by applying a sliding window variance algorithm to the throughput indicator; Furthermore, the sampling period is inversely proportional to the degree of fluctuation.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the feature type of the migration task, wherein the feature type includes at least one of cold data, hot data, and mixed data; Configure at least one of the proportional coefficient, integral coefficient, and derivative coefficient of the proportional-integral-derivative control according to the feature type; The proportion coefficient of cold data migration tasks is higher than that of hot data migration tasks, and the integral coefficient of cold data migration tasks is lower than that of hot data migration tasks.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images