Data migration control method, device, storage medium, and program product

Abstract

Embodiments of the present application provide a data migration control method and device, a storage medium and a program product, and relate to the technical field of databases. The method comprises: reading data from a source end and storing the data in a read buffer according to a back pressure control instruction; converting and transmitting the data in the read buffer to a write buffer; writing the data in the write buffer to a target end; and in the reading, transmitting and writing of the data, the following operations are synchronously performed: decoupling adjustment of the capacities of the read buffer and the write buffer; and updating the back pressure control instruction according to a monitoring result of a state parameter of the write buffer and a performance state of the target end, to adjust subsequent data reading operations. The method of the present application increases the cooperative mechanism of the decoupling adjustment of the double buffers and the back pressure control based on the multi-level strategy, and improves the stability and throughput efficiency of large-scale data migration tasks.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data migration control method, device, storage medium, and program product. Background Technology

[0002] In the era of big data, scenarios such as enterprise-level database migration and data warehouse construction are becoming increasingly common, with the core requirement being the migration of existing data. These migration tasks typically involve massive amounts of data, have long migration cycles, and place high demands on system stability and performance. Therefore, achieving reliable and smooth migration of large-scale data has become a pressing technical problem to be solved in enterprise information transformation.

[0003] In current implementations, traditional data migration tools based on batch processing or queuing mechanisms are typically used to accomplish this type of task. Common implementation methods include: extracting data from the source database in batches and temporarily storing it in an intermediate buffer, and then writing it to the target database in batches; or using a producer-consumer model to build a data pipeline and perform read and write operations at a fixed rate.

[0004] However, the above method is prone to memory overflow or system crash due to the continuous accumulation of data in the intermediate buffer. Summary of the Invention

[0005] This application provides a data migration control method, device, storage medium, and program product to solve the technical problem of memory overflow or system crash caused by continuous data accumulation in the buffer during the migration process.

[0006] In a first aspect, embodiments of this application provide a data migration control method applied to a migration control terminal. The migration control terminal is communicatively connected to both a source terminal and a target terminal, and maintains mutually independent read buffers and write buffers. The method includes:

[0007] Data is read from the source and stored in the read buffer according to the back pressure control command, where the back pressure control command is used to indicate the batch size of the read data or control the execution status of the read operation;

[0008] The data in the read buffer is transformed and then passed to the write buffer;

[0009] Write the data in the write buffer to the target end;

[0010] During the reading, transmission, and writing of data, the following operations are performed synchronously:

[0011] The capacities of the read buffer and the write buffer are decoupled and adjusted. The capacity of the read buffer is adjusted based on the read performance parameters of the source end, and the capacity of the write buffer is adjusted based on the write performance parameters of the target end.

[0012] Furthermore, based on the monitoring results of the write buffer status parameters and the target end performance status, the back pressure control instructions are updated with a multi-level control strategy to adjust subsequent data read operations.

[0013] In this embodiment, by using a decoupled and adjusted dual-buffer architecture and a dynamic backpressure mechanism based on a multi-level control strategy, an adaptive balance between the data inflow rate and the target processing capacity is achieved during the data migration process. This solves the technical problems in large-scale data migration scenarios, such as write buffer data backlog, memory overflow, system crash, and low resource utilization caused by mismatch between read and write rates, thereby improving the stability, throughput efficiency, and resource utilization of the data migration task.

[0014] In one possible implementation, the capacities of the read buffer and the write buffer are decoupled and adjusted, including:

[0015] When the read buffer occupancy rate reaches the first high threshold, the read buffer is expanded; when the read buffer occupancy rate reaches the first low threshold, the read buffer is shrunk.

[0016] When the write buffer occupancy rate reaches the second highest threshold, the write buffer is expanded; when the write buffer occupancy rate reaches the second lowest threshold, the write buffer is shrunk.

[0017] In this implementation, through the aforementioned mechanism, this application achieves decoupled dynamic adjustment of the read buffer and write buffer capacities. The read buffer expands and shrinks independently based on the source end's read performance, effectively absorbing read rate fluctuations and preventing data backlog at the source end; the write buffer expands and shrinks independently based on the target end's write performance, providing sufficient temporary buffer space when the target end's processing capacity decreases, preventing data overflow. While ensuring stable system operation and avoiding memory overflow and migration interruptions, this approach improves buffer resource utilization and data migration throughput efficiency.

[0018] In one possible implementation, the backpressure control command is updated using a multi-level control strategy, including:

[0019] In response to monitoring results meeting different conditions, the back pressure control command is updated to a different type of command;

[0020] The different types of instructions include a first type of instructions for adjusting the batch size of reads, and a second type of instructions for controlling the start and stop of read operations.

[0021] In this implementation, the back pressure controller achieves gradient control of the reading module through the aforementioned multi-level control strategy. When the system pressure is slight, rate regulation control is prioritized to avoid system disturbances caused by frequent start-stop operations; when the system pressure is severe, it promptly switches to start-stop control protection to ensure that the system does not collapse due to overload, thereby improving system stability.

[0022] In one possible implementation, the source-side read performance parameters include at least one of the data fetch rate and read operation latency;

[0023] The target end's write performance parameters include at least one of the following: data write rate, transaction processing rate, and operation latency.

[0024] In this implementation, through the above-mentioned performance parameter configuration, this application achieves accurate quantification and real-time perception of the source end's read capability and the target end's write capability, providing an accurate decision basis for the decoupling adjustment of the read buffer and the write buffer, thereby further improving the accuracy of dynamic adaptation of the buffer capacity.

[0025] In one possible implementation, decoupling the capacity adjustments of the read buffer and the write buffer also includes:

[0026] Based on historical load time-series data from the source and destination ends, predictive calculations are performed to estimate future load demands on the read and write buffers.

[0027] Based on the results of predictive calculations, capacity adjustment strategies for the read buffer and write buffer are determined separately, and adjustments are made based on these strategies.

[0028] In this implementation, through the aforementioned predictive adjustment mechanism, the buffer manager elevates buffer capacity management from reactive response to proactive pre-adjustment, resolving the response lag issue inherent in real-time threshold-triggered methods. Predictive adjustment and real-time responsive adjustment based on water level thresholds complement and coordinate with each other, jointly forming a multi-time-dimensional adaptive buffer capacity management system.

[0029] In one possible implementation, the method further includes:

[0030] Based on the data's priority attribute, assign scheduling priority to the data;

[0031] When selecting data from the read buffer to transfer to the write buffer, and when selecting data from the write buffer to write to the target, data with higher priority is selected for transfer or writing operations based on the scheduling priority.

[0032] In this implementation, through the aforementioned priority scheduling mechanism, this application ensures overall migration throughput efficiency while providing differentiated quality of service (QoS) guarantees for critical business data. This mechanism is decoupled from and coexists with the dynamic dual-buffer architecture, hierarchical backpressure control, and predictive scaling strategies. Users can flexibly enable or disable it according to business needs, achieving a unification of general migration capabilities and dedicated service guarantees.

[0033] In one possible implementation, the method further includes:

[0034] In response to the data volume in the write buffer exceeding the maximum memory limit allocated to the write buffer, a portion of the data in the write buffer is transferred to disk storage.

[0035] In this implementation, through the above mechanism, this application extends the capacity boundary of the write buffer from physical memory to disk storage, thereby improving the system's fault tolerance and business continuity assurance level in the face of extreme scenarios such as instantaneous write peaks and long-term unavailability of the target end without increasing hardware costs.

[0036] In a second aspect, this application provides an electronic device, including: a processor and a memory communicatively connected to the processor;

[0037] The memory stores instructions that the computer executes;

[0038] The processor executes computer-executable instructions stored in memory to implement any of the methods of the first aspect.

[0039] Thirdly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of the first aspects.

[0040] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the first aspects. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] Figure 1 A flowchart illustrating a data migration control method provided in an embodiment of this application;

[0043] Figure 2 This application provides a schematic diagram of the data flow of a data migration system according to an embodiment of the present application.

[0044] Figure 3This is a flowchart illustrating a multi-dimensional decision-making process for a back pressure controller, as provided in an embodiment of this application.

[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] It should be noted that the data migration control method, device, storage medium and program product provided in this application can be used in the field of database technology, or in any field other than database technology. This application does not limit the application field of the data migration control method, device, storage medium and program product.

[0048] This application can be applied to various enterprise-level applications requiring large-scale, long-cycle, and highly reliable data migration, especially in scenarios where there are significant differences in data processing capabilities between the source and target systems. Specific scenarios include, but are not limited to, the following:

[0049] Migrating traditional on-premises databases to cloud databases: For example, migrating locally deployed relational databases such as Oracle and MySQL to cloud database services. The migration process involves the complete and consistent transfer of tens of TB to PB of historical data.

[0050] The data warehouse extraction-transformation-loading process: When building or updating an enterprise data warehouse, it is necessary to extract massive amounts of data from multiple heterogeneous business systems and achieve cross-system and cross-structure data synchronization and integration during the transformation process.

[0051] Real-time data replication and disaster recovery across data centers: For example, in industries with high business continuity requirements such as finance and telecommunications, it is necessary to achieve near real-time data synchronization between production centers and disaster recovery centers to ensure data integrity and availability during failover.

[0052] In these scenarios, migration tasks typically share the following characteristics: large data scale, long migration cycle (ranging from hours to days), and significant performance asymmetry between the source and target systems in reading, writing, or network transmission. For example, the source may have sufficient network bandwidth while the target may have a significantly lower write rate than read rate due to index maintenance, transaction lock contention, or storage I / O bottlenecks. This can easily lead to data backlog and decreased system stability during the migration process.

[0053] To address the aforementioned application scenarios and meet the demands of large-scale data migration, existing technical solutions typically employ the following three main approaches:

[0054] One possible implementation is a migration approach based on large batch loading and batch processing. This approach uses fixed-size data batches as the basic processing unit for migration. The specific process is as follows: First, the migration tool continuously reads data from the source database according to a preset fixed number of rows (e.g., every 10,000 rows as a batch) and loads the entire batch of data completely into the application's memory. Then, the batch of data undergoes format conversion, cleaning, or calculations in memory. Finally, the processed batch of data is submitted to the target database for writing in a single batch operation. The entire migration process consists of a series of sequentially executed batch processing tasks, and its processing pace and resource consumption are mainly determined by the preset batch size.

[0055] In another possible implementation, a migration approach based on multithreading and fixed queues could be used. This approach employs a producer-consumer model to build an asynchronous processing pipeline. Specifically, the system creates at least one data reading thread as a producer, which continuously extracts data from the source and places it into a buffer queue with a pre-configured fixed capacity (e.g., a BlockingQueue with an upper limit).

[0056] Simultaneously, the system creates at least one data writing thread as a consumer. This thread retrieves data from the other end of the buffer queue and performs the write operation to the target end. Through multi-threaded parallel operation and buffering of the intermediate queue, this scheme aims to improve data processing throughput and alleviate the direct coupling between read and write operations.

[0057] Another possible implementation is a migration approach based on static rate limiting. This approach imposes a constant rate cap on the data migration path to control the overall migration traffic. Specifically, the system typically integrates a rate controller into the data read or write module. This controller regulates the data throughput based on a pre-set threshold that remains constant throughout the migration cycle (e.g., a maximum of 1000 records per second). By limiting the amount of data processed per unit time, this approach attempts to prevent performance degradation or service unavailability of the target system due to short-term request overload.

[0058] However, the solution described above has the following technical problems:

[0059] First, in schemes based on fixed batch loading and batch processing, the pre-set batch size cannot adapt to dynamic changes in data characteristics. When the amount of data in a single batch increases due to differences in record length or field content, the system memory usage will experience sudden peaks, which can easily trigger the risk of memory overflow, thereby affecting the stability of long-running migration tasks.

[0060] Secondly, while the static queue and multi-threaded parallel approach introduces a buffering mechanism, its queue capacity is typically fixed and cannot be adjusted according to real-time changes in the target system's write performance. When the target system experiences a slowdown in write speed due to index maintenance, transaction lock contention, or resource bottlenecks, the read end will continue to fill the already full or nearly full queue with data, eventually leading to system resource exhaustion due to excessive accumulation of buffered data.

[0061] Furthermore, while static rate limiting schemes can provide basic flow control, their preset fixed rate thresholds are difficult to respond to factors such as network fluctuations and changes in target library load during the migration process. This results in unnecessary limitation of migration throughput when the target system has sufficient processing capacity, and an inability to reduce the read rate in time when the target system experiences temporary performance degradation, causing data to accumulate in the pipeline, thus affecting the overall migration efficiency and reliability.

[0062] Furthermore, existing solutions generally lack the ability to coordinate and adaptively control all links in the process (reading, conversion, and writing), making it difficult to achieve pipeline-style dynamic balancing based on the actual processing speed at each stage. This fixed configuration of processing capabilities means that when faced with long-term, large-scale migration tasks, the system cannot fully improve resource utilization, nor can it guarantee the continuous and stable operation of the migration process.

[0063] The data migration control method provided in this application aims to solve the aforementioned technical problems of existing technologies. By employing an independently and dynamically adjustable dual-buffer structure, combined with a multi-level hierarchical backpressure feedback mechanism, a balance is achieved between the processing rates of the read and write ends during data migration, thereby ensuring the system stability of large-scale, long-cycle migration tasks. The specific implementation includes the following key aspects:

[0064] First, a dual-buffer structure capable of independent dynamic expansion and contraction is constructed, specifically comprising a read buffer and a write buffer. This structure allows the buffer capacity to be flexibly adjusted based on their respective driving factors: the read buffer adjusts according to changes in the data read rate at the source end, and the write buffer adjusts according to changes in the data write rate at the destination end. This avoids resource waste or buffer overflow that might occur when using a single fixed buffer during data flow fluctuations.

[0065] Secondly, a tiered backpressure mechanism was designed and implemented. This mechanism generates multi-level backpressure control signals based on real-time monitoring of the target end's write speed, assessment of the target system's load status, and comprehensive analysis of the current water level of each buffer. These signals are fed back to the data reading end to adjust the data extraction rate in real time. This suppresses upstream data supply when the target end's processing capacity is insufficient or the buffer is approaching saturation, preventing the system from overloading or crashing due to data backlog.

[0066] Furthermore, the data migration process is organized into a pipeline consisting of three stages: reading, transformation, and writing, with an interlocking coordination mechanism introduced between each stage. This mechanism ensures that the processing capabilities of each stage are matched, preventing a bottleneck in one stage from causing subsequent stages to be idle or the entire process to be blocked, thereby improving the resource utilization and task throughput of the entire chain.

[0067] In summary, this application achieves adaptive adjustment of performance fluctuations during the migration process through the synergistic effect of the aforementioned dynamic buffer management, hierarchical backpressure control, and pipeline interlocking. This improves data migration efficiency while ensuring the stability and reliability of the system during long-term operation.

[0068] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0069] First, combine Figures 1 to 3 The data migration control method provided in the embodiments of this application will be introduced. Figure 1 This is a flowchart illustrating a data migration control method provided in an embodiment of this application. Figure 2 This is a schematic diagram of the data flow of a data migration system provided in an embodiment of this application.Figure 3 This is a flowchart illustrating a multi-dimensional decision-making process for a back pressure controller, as provided in an embodiment of this application.

[0070] It should be noted that, in this application's embodiments, the migration control terminal refers to a logical entity or physical device deployed independently of the source and target ends, used for centrally executing data migration tasks and implementing dynamic flow control and resource management. The migration control terminal can be a standalone server, a control node in a distributed system, or a software module embedded in a data migration tool; this application's embodiments do not impose any limitations on this. The migration control terminal communicates with both the source and target ends, and is responsible for coordinating the collaborative work of components such as the read module, conversion module, write module, buffer manager, and backpressure controller.

[0071] like Figure 1 As shown, the method includes the following steps:

[0072] S101: Read data from the source end according to the back pressure control command and store it in the read buffer.

[0073] The read module in the migration control unit reads data from the source in batches according to the currently effective backpressure control command. The backpressure control command is a control signal dynamically generated by the backpressure controller in the migration control unit, used to indicate the batch size of the read operation, or directly control the execution status of the read operation, such as start, pause, or stop. The read raw data is stored in the read buffer.

[0074] S102. Convert the data in the read buffer and transfer it to the write buffer.

[0075] The conversion module in the migration control unit retrieves data from the read buffer and performs preprocessing operations such as format conversion, type validation, and data cleaning to ensure that the data format and content meet the storage requirements of the target end. The converted data is then temporarily stored in the write buffer, awaiting subsequent writing.

[0076] S103. Write the data in the write buffer to the target end.

[0077] The write module in the migration control terminal retrieves the converted data from the write buffer and writes it to the target terminal, completing the write operation of the data migration.

[0078] S104. Decouple and adjust the capacity of the read buffer and the write buffer.

[0079] The buffer manager in the migration control unit performs independent and decoupled dynamic adjustments to the capacity of the read buffer and the write buffer. Decoupled adjustment means that the capacity adjustment decisions for the read buffer and the write buffer are independent of each other, based on their respective performance indicators, and do not interfere with each other.

[0080] The capacity adjustment of the read buffer is based on the read performance parameters of the source end. Read performance parameters are quantitative indicators reflecting the data read capability of the source end, and include at least one of the following: data fetch rate and read operation latency. Data fetch rate refers to the amount of data read from the source end per unit time, and read operation latency refers to the time required from initiating a read request to the return of the completed data.

[0081] The write buffer capacity is adjusted based on the target's write performance parameters. Write performance parameters are quantitative indicators reflecting the target's data write capability, and include at least one of the following: data write rate, transaction processing rate, and operation latency. Data write rate refers to the amount of data successfully written to the target per unit of time; transaction processing rate refers to the number of transactions the target can process per second; and operation latency refers to the time required from initiating a write request to confirming the write completion.

[0082] S105. Based on the monitoring results of the status parameters of the write buffer and the performance status of the target end, update the back pressure control instruction with a multi-level control strategy to adjust the subsequent data reading operation.

[0083] The backpressure controller in the migration control unit continuously monitors the status parameters of the write buffer and the performance status of the target end. The write buffer status parameters are quantitative indicators reflecting the current load status of the write buffer, including at least the data backlog and occupancy rate. The data backlog refers to the total amount of data currently waiting to be written to the target end in the write buffer, while the occupancy rate is the ratio of the used capacity of the write buffer to its total capacity. The target end's performance status is a comprehensive indicator reflecting its current operating load and processing capacity, including at least CPU utilization, memory utilization, I / O load, and transaction queue length.

[0084] Based on monitoring results, the back pressure controller dynamically updates the back pressure control commands sent to the reading module according to a preset multi-level control strategy, thereby adjusting the batch size or execution status of subsequent data reading operations. The multi-level control strategy refers to dividing the system pressure into different levels, with each level corresponding to different control strengths and actions, to achieve gradient back pressure control from mild current limiting to severe protection.

[0085] The following combination Figure 2 The data flow of the data migration system in the embodiments of this application will be described in detail. Figure 2 The content shown corresponds to Figure 1 Steps S101 to S103 are shown.

[0086] like Figure 2 As shown, the data migration system includes a read module, a read buffer, a conversion module, a write buffer, and a write module. These modules are connected in series in the order of "read → read buffer → conversion → write buffer → write," forming the core data flow path for data migration.

[0087] The read module is the execution unit responsible for retrieving data from the source. The read buffer is memory space used to temporarily store the raw data retrieved by the read module. The conversion module is the processing unit responsible for preprocessing operations such as format conversion and content cleaning on the raw data. The write buffer is memory space used to temporarily store the data to be written after being processed by the conversion module. The write module is the execution unit responsible for writing the data in the write buffer to the target end.

[0088] The data flow of each module is as follows: the reading module reads data from the source and stores the data in the read buffer; the conversion module obtains data from the read buffer; the conversion module completes the data conversion and then passes the data to the write buffer; and the writing module obtains data from the write buffer and writes it to the target end.

[0089] The read buffer and write buffer are logically and physically independent, and their capacity is managed by the buffer manager. The read module, conversion module, and write module are all subject to coordinated control by the backpressure controller and the buffer manager while executing the main data migration process.

[0090] The following combination Figure 3 The multi-dimensional decision-making logic of the back pressure controller in the embodiments of this application is described in detail. Figure 3 Corresponding to Figure 1 Steps S104 to S105 are shown. Figure 3 As shown, the decision logic of the back pressure controller includes two parts that work together: the main process on the left and the auxiliary process on the right.

[0091] First, let's introduce the main process on the left, which represents the core decision-making logic of the back pressure controller.

[0092] The backpressure controller receives real-time status information reported by the write module. The status information reported by the write module includes at least the write speed, Transactions Per Second (TPS), operation latency, and error rate. Write speed refers to the amount of data successfully written to the target device per unit of time; TPS refers to the number of transaction commits completed by the write module per unit of time; operation latency refers to the time required from the write module initiating a write request to the target device returning a write confirmation; and the error rate refers to the proportion of failed write operations to the total number of write operations.

[0093] The backpressure controller continuously monitors both the write buffer occupancy status and the resource load on the target side. Write buffer occupancy status refers to the ratio of the currently used write buffer capacity to its total capacity. Target-side resource load includes at least CPU utilization, I / O load, and memory usage.

[0094] Based on the above multi-dimensional monitoring data, the back pressure controller performs the following three types of judgments in parallel:

[0095] First, determine if the write end has slowed down. When the write speed decreases, TPS decreases, operation latency increases, or the error rate increases, it is determined that a performance bottleneck has occurred on the write end. In response to this determination, the first-level backpressure strategy is activated, and at least one of the following rate-limiting operations is executed: reducing the read batch size, reducing the number of concurrent read threads, or reducing the number of processing threads.

[0096] Second, determine if the target's CPU or I / O load is too high. When the target's CPU utilization, I / O load, or memory usage exceeds a preset threshold, it is determined that the target is at risk of resource exhaustion. In response to this determination, a protection mode is activated, and at least one of the following operations is performed: pausing reads, further reducing the read batch size, or reducing the number of concurrent threads.

[0097] Third, determine if the write buffer has reached the high-water mark. When the write buffer occupancy rate reaches or exceeds the preset high-water mark threshold, a buffer overflow risk is identified. In response to this determination, buffer protection is activated, performing at least one of the following operations: pausing reads, reducing write speed, or waiting for the buffer to be emptied.

[0098] When any of the above conditions is triggered, the backpressure controller immediately sends an updated backpressure control command to the reading module to adjust the batch size or execution status of subsequent data reading operations. This command update process corresponds to... Figure 1 The step S105 shown.

[0099] Secondly, the auxiliary process on the right is introduced, which represents the backend load sampling and periodic adjustment.

[0100] The system samples the load status of the target device at preset intervals. The sampling metrics include CPU utilization, I / O load, memory usage, and buffer usage. The sampling results are then compared with preset security thresholds.

[0101] If the backend load does not exceed the safety threshold, the system maintains the current processing rate, and the reading module and the conversion module continue to perform data reading and conversion operations according to the existing parameters.

[0102] If the backend load exceeds the safety threshold, the backpressure decision loop of the main process on the left is triggered. Based on the decision result, the number of concurrent threads of the reading module, the size of the reading batch, and the processing speed of the conversion module are dynamically adjusted to form a complete closed-loop control of "monitoring-judgment-backpressure-adjustment".

[0103] The data migration control method provided in this embodiment achieves an adaptive balance between the data inflow rate and the target processing capacity during the data migration process through a decoupled and adjusted dual-buffer architecture and a dynamic backpressure mechanism based on a multi-level control strategy. This solves the technical problems such as write buffer data backlog, memory overflow, system crash, and low resource utilization caused by read-write rate mismatch in large-scale data migration scenarios, thereby improving the stability, throughput efficiency, and resource utilization of the data migration task.

[0104] Optionally, the operation of decoupling and adjusting the capacity of the read buffer and the write buffer in the above embodiment is specifically implemented as follows.

[0105] The buffer manager configures a first high threshold and a first low threshold for the read buffer, and a second high threshold and a second low threshold for the write buffer. These thresholds are set as a percentage of buffer occupancy. For example, the first high threshold can be set to 80%, meaning that an expansion condition is triggered when the read buffer occupancy reaches 80%; the first low threshold can be set to 20%, meaning that a shrinking condition is triggered when the read buffer occupancy drops to 20%. The second high threshold can be set to 75%, and the second low threshold can be set to 25%.

[0106] It should be noted that the specific values ​​mentioned above are merely illustrative examples and do not constitute a limitation on the scope of protection of this application. In practical applications, they can be dynamically configured or adaptively adjusted according to system memory resources, data traffic characteristics, and target processing capabilities.

[0107] The buffer manager continuously monitors the current occupancy rate of the read buffer and compares it with a first high threshold and a first low threshold.

[0108] When the read buffer occupancy rate reaches or exceeds the first high threshold, it indicates that the read buffer is about to become saturated. At this point, the buffer manager determines that the read buffer needs to be expanded and immediately performs the expansion operation to increase the capacity allocation of the read buffer.

[0109] When the read buffer occupancy rate drops to or below the first low threshold, it indicates that the read buffer is currently under light load and there are idle memory resources. At this time, the buffer manager determines that the read buffer can be shrunk and immediately executes the shrinking operation to reclaim some capacity to free up system memory.

[0110] Similarly, the buffer manager uses the same monitoring and judgment logic for the write buffer. When the occupancy rate reaches or exceeds the second highest threshold, an expansion operation is performed; when the occupancy rate drops to or falls below the second lowest threshold, a shrinkage operation is performed.

[0111] In this way, the buffer manager achieves decoupled adjustment of the read buffer and write buffer capacities based on water level thresholds. This adjustment process is executed synchronously with the main data migration flow of steps S101 to S103, and works in conjunction with the back pressure control command update mechanism in step S105 to form a complete adaptive data migration control system.

[0112] Through the above mechanism, this application achieves decoupled dynamic adjustment of the read buffer and write buffer capacities. The read buffer expands and shrinks independently based on the source end's read performance, effectively absorbing read rate fluctuations and preventing data backlog at the source end; the write buffer expands and shrinks independently based on the target end's write performance, providing sufficient temporary buffer space when the target end's processing capacity decreases, preventing data overflow. While ensuring stable system operation and avoiding memory overflow and migration interruptions, it improves buffer resource utilization and data migration throughput efficiency.

[0113] Optionally, in this embodiment of the application, the specific implementation of the back pressure controller performing multi-level strategy updates on the back pressure control command is as follows.

[0114] The backpressure controller continuously monitors the status parameters of the write buffer and the performance status of the target device. Based on the monitoring results, the backpressure controller updates the currently effective backpressure control commands to different types of commands according to a preset multi-level control strategy to adapt to different levels of system pressure. The multi-level control strategy is divided into multiple levels according to the severity of system pressure, with each level corresponding to different monitoring conditions and command types.

[0115] The first type of instruction is used to adjust the batch size of the read module. When the write buffer occupancy rate is detected to be rising continuously but not yet reaching a critical value, or when the write speed at the target end shows a slight decrease, the backpressure controller determines that the system is under slight pressure, generates the first type of instruction, and sends it to the read module. The first type of instruction carries the adjusted batch size parameter. Based on this instruction, the read module reduces the amount of data read in a single operation from the current value to the target value, for example, from 5000 lines to 3000 lines, thereby reducing the data production rate and alleviating the downstream processing pressure.

[0116] The second type of instruction controls the start and stop status of read operations. When the write buffer occupancy rate exceeds the high-water mark threshold, the target CPU utilization rate remains above the load threshold, or the target write error rate increases significantly, the backpressure controller determines that the system is under heavy pressure, generates the second type of instruction, and sends it to the read module. The second type of instruction carries a pause execution flag. Upon receiving this instruction, the read module immediately pauses the data read operation. When the system pressure subsequently eases, the write buffer begins to empty, and the target performance returns to normal, the backpressure controller generates a second type of instruction carrying a resume execution flag, and the read module resumes data reading accordingly.

[0117] In some implementation scenarios, multi-level control strategies may also include intermediate levels between the two types of instructions mentioned above. For example, when the system pressure is between light and heavy, the back pressure controller may generate instructions to reduce the number of concurrent read threads, or generate compound instructions to adjust both the batch size and the number of concurrent threads.

[0118] Through the aforementioned multi-level control strategy, the backpressure controller achieves gradient regulation of the reading module. When the system pressure is slight, rate regulation control is prioritized to avoid system disturbances caused by frequent start-stop operations; when the system pressure is severe, it promptly switches to start-stop control protection to ensure that the system does not collapse due to overload, thereby improving system stability.

[0119] Optionally, the specific composition of the read performance parameters of the source end and the write performance parameters of the target end in this embodiment of the application is as follows.

[0120] Read performance parameters include at least one of data fetch rate and read operation latency, and write performance parameters include at least one of data write rate, transaction processing rate, and operation latency.

[0121] The specific definitions, collection methods, and applications of the above indicators in buffer capacity adjustment can be understood by referring to the relevant description in step S104, and will not be repeated here.

[0122] It should be noted that the performance parameters mentioned above in this application are merely illustrative examples and not exhaustive. In practical applications, other equivalent indicators that can reflect the source end's read capability or the target end's write capability can be added or replaced according to the specific business scenario and system environment.

[0123] By configuring the above performance parameters, this application achieves accurate quantification and real-time perception of the source end's read capability and the target end's write capability, providing an accurate decision basis for decoupling and adjusting the read buffer and write buffer, thereby further improving the accuracy of dynamic adaptation of buffer capacity.

[0124] Optionally, based on the real-time responsive scaling up and down based on water level thresholds, this embodiment further introduces a predictive adjustment mechanism to achieve dynamic management of the read buffer and write buffer capacity.

[0125] The buffer manager continuously collects historical load time-series data from both the source and target ends. The historical load time-series data from the source end includes at least the data fetch rate and read operation latency of the read module over a past period; the historical load time-series data from the target end includes at least the data write rate, transaction processing rate, write operation latency, and write buffer occupancy trend of the write module over a past period.

[0126] Based on the aforementioned historical load time-series data, the buffer manager employs a preset prediction algorithm to predict the future load demands of the read and write buffers. The prediction algorithm can use time series analysis, moving averages, exponential smoothing, or linear regression methods to estimate the source read rate, target write rate, and buffer occupancy rate within one or more future time windows.

[0127] Taking the write buffer as an example, when it is predicted that the write rate of the target end will continue to decline within the next 30 seconds, or that the write buffer occupancy rate will reach the high watermark threshold within the next 60 seconds, the buffer manager does not need to wait for the real-time threshold to be triggered, but instead initiates an expansion operation to increase the write buffer capacity and absorb the upcoming data backlog pressure. When the prediction results indicate that the write rate of the target end will steadily recover over a period of time, the buffer manager initiates a shrinkage operation in advance to gradually reclaim excess memory resources.

[0128] For the read buffer, when it is predicted that the source read rate will increase significantly during peak business hours, the buffer manager will expand the read buffer in advance; when it is determined that the source read rate will enter a low period, it will shrink the buffer in advance.

[0129] Predictive calculation results and real-time monitoring data are used together in capacity adjustment decisions. In some implementation scenarios, the buffer manager sets an independent confidence threshold for predictive adjustments, triggering the pre-adjustment operation only when the prediction result reaches a certain level of confidence, thus avoiding ineffective expansion or contraction due to prediction bias.

[0130] Through the aforementioned predictive adjustment mechanism, the buffer manager elevates buffer capacity management from reactive response to proactive pre-adjustment, resolving the response lag issue inherent in real-time threshold-triggered methods. Predictive adjustment and real-time responsive adjustment based on water level thresholds complement and coordinate with each other, forming a multi-time-dimensional adaptive buffer capacity management system.

[0131] Optionally, this embodiment further introduces a priority-based scheduling mechanism to achieve priority processing of critical business data.

[0132] The migration control panel configures priority attributes for the data to be processed. Priority attributes are levels of information that identify the importance or timeliness requirements of data. They can be derived from preset priority fields in the source data table, or dynamically identified and marked by the migration control panel according to business rules. For example, core transaction records can be marked as high priority, while log backups or historical archived data can be marked as low priority.

[0133] When the conversion module selects data from the read buffer to transfer to the write buffer, it schedules the data according to their priority. The conversion module prioritizes selecting data with higher priority from the read buffer for format conversion and transfer, ensuring that high-priority data enters the write buffer and is written to the target device earlier. Low-priority data is only processed if there is no high-priority data in the read buffer or if high-priority data has been processed.

[0134] When the write module selects data from the write buffer to write to the target, it also schedules the data according to its scheduling priority. The write module prioritizes selecting data with higher scheduling priority from the write buffer to perform write operations, minimizing the time that high-priority data stays in the write buffer and reducing end-to-end migration latency. When the write buffer faces backlog pressure, low-priority data may be temporarily deferred to make way for high-priority data, freeing up buffer space and write bandwidth.

[0135] The priority scheduling mechanism can work in conjunction with the backpressure control mechanism. When the backpressure controller sends a deceleration or pause command to the reading module due to excessive load on the target end, the priority scheduling mechanism ensures that, within the limited migration throughput capacity, high-priority data can still be processed first, while low-priority data is appropriately degraded or delayed.

[0136] Through the aforementioned priority scheduling mechanism, this application ensures overall migration throughput efficiency while providing differentiated quality of service (QoS) guarantees for critical business data. This mechanism is decoupled from the dynamic dual-buffer architecture, hierarchical backpressure control, and predictive scaling strategies, allowing users to flexibly enable or disable it according to business needs, thus unifying general migration capabilities with dedicated service guarantees.

[0137] Optionally, this embodiment further introduces a memory-disk hybrid buffer mechanism to address the risk of memory resource exhaustion under extreme load scenarios.

[0138] The write buffer is allocated a fixed maximum memory capacity during initialization. This maximum capacity is set by the buffer manager based on the system's available memory resources and the scale of the migration task. Under normal operating conditions, the write buffer only temporarily stores data to be written in memory to ensure maximum read and write performance.

[0139] The buffer manager continuously monitors the current amount of data in the write buffer and compares the monitored amount of data with the maximum memory limit allocated to that write buffer.

[0140] When the amount of data in the write buffer reaches or exceeds the memory limit, it indicates that the memory buffer resources are about to be exhausted. If data continues to be written to the write buffer, it will trigger a memory overflow error. At this time, the buffer manager activates the overflow protection mechanism, transferring part of the data in the write buffer from memory to disk storage.

[0141] The transfer operation follows a first-in, first-out (FIFO) or low-priority-to-high-priority order, selecting data temporarily stored in the write buffer and writing it to a pre-allocated temporary disk file. After the transfer is complete, the memory space occupied by the corresponding data is released, and the write buffer regains available memory capacity, enabling it to continue receiving new data from the conversion module.

[0142] The data transferred to disk does not leave the migration process. When the memory usage of the write buffer falls below a safe threshold, the buffer manager can reload the data temporarily stored on the disk into memory and continue waiting for the write module to write it to the target. In some implementation scenarios, the write module can also directly read data from the temporary disk file and perform the write operation, avoiding repeated data migration between memory and disk.

[0143] The memory-disk hybrid buffer mechanism and the backpressure control mechanism form a dual protection. Backpressure control first alleviates the pressure on the write buffer by reducing the data production rate; when the backpressure adjustment has reached its limit or the load increases too quickly to take effect, the hybrid buffer mechanism acts as the last line of defense, avoiding system crashes by borrowing space from the disk.

[0144] Through the above mechanism, this application extends the capacity boundary of the write buffer from physical memory to disk storage, thereby improving the system's fault tolerance and business continuity assurance level in extreme scenarios such as instantaneous write peaks and long-term unavailability of the target end without increasing hardware costs.

[0145] The electronic device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0146] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the methods in any of the above method embodiments.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods in any of the above method embodiments.

[0148] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0149] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes ​ The steps of the function specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0153] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0154] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0155] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0156] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0157] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0158] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0159] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data migration control method, characterized in that, The method, applied to a migration control terminal that is communicatively connected to both the source and target terminals and maintains independent read and write buffers, includes: Data is read from the source end according to the back pressure control command and stored in the read buffer, wherein the back pressure control command is used to indicate the batch size of the read data or control the execution status of the read operation; The data in the read buffer is converted and then transferred to the write buffer; Write the data in the write buffer to the target terminal; During the reading, transmission, and writing of the data, the following operations are performed synchronously: The capacities of the read buffer and the write buffer are decoupled and adjusted, wherein the capacity adjustment of the read buffer is based on the read performance parameters of the source end, and the capacity adjustment of the write buffer is based on the write performance parameters of the target end. Furthermore, based on the monitoring results of the state parameters of the write buffer and the performance status of the target end, the back pressure control command is updated with a multi-level control strategy to adjust subsequent data reading operations.

2. The method according to claim 1, characterized in that, The decoupling adjustment of the capacities of the read buffer and the write buffer includes: In response to the occupancy rate of the read buffer reaching a first high threshold, an expansion operation is performed on the read buffer; in response to the occupancy rate of the read buffer reaching a first low threshold, a shrinking operation is performed on the read buffer. In response to the write buffer occupancy rate reaching a second high threshold, an expansion operation is performed on the write buffer; in response to the write buffer occupancy rate reaching a second low threshold, a shrinking operation is performed on the write buffer.

3. The method according to claim 2, characterized in that, The step of updating the back pressure control command with a multi-level control strategy includes: In response to different conditions being met by the monitoring results, the back pressure control command is updated to a different type of command; The different types of instructions include a first type of instruction for adjusting the batch size of reads, and a second type of instruction for controlling the start and stop of read operations.

4. The method according to claim 3, characterized in that, The source end's read performance parameters include at least one of data fetch rate and read operation latency; The write performance parameters of the target end include at least one of the following: data write rate, transaction processing rate, and operation latency.

5. The method according to claim 1 or 2, characterized in that, The decoupling adjustment of the capacities of the read buffer and the write buffer also includes: Based on the historical load time series data of the source and target ends, predictive calculations are performed on the future load requirements of the read buffer and the write buffer. Based on the results of the predictive calculation, capacity adjustment strategies for the read buffer and the write buffer are determined respectively, and adjustments are made based on the capacity adjustment strategies.

6. The method according to claim 1, characterized in that, The method further includes: Based on the data's priority attribute, assign scheduling priority to the data; When selecting data from the read buffer to transfer to the write buffer, and when selecting data from the write buffer to write to the target end, data with higher priority is selected for the transfer or write operation based on the scheduling priority.

7. The method according to claim 1 or 2, characterized in that, The method further includes: In response to the data volume in the write buffer exceeding the memory limit allocated to the write buffer, a portion of the data in the write buffer is transferred to disk storage.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images