Data migration method and related equipment based on queue water level
By using a dynamic task allocation and theft mechanism based on queue levels, the problem of task backlog and reduced throughput in existing parallel migration systems under massive small file scenarios is solved, achieving efficient and stable data migration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2026-03-16
- Publication Date
- 2026-06-30
AI Technical Summary
Existing parallel migration systems, in scenarios with massive amounts of small files, lack closed-loop flow control based on queue levels. When the production rate far exceeds the consumption rate, tasks accumulate without bounds, triggering GC and OOM, resulting in migration interruptions. When the consumer suddenly becomes idle, it cannot wake up production in time, causing a sharp drop in overall throughput, forming a vicious cycle, and ultimately causing migration failure.
By using a data migration method based on queue water level, the task allocation speed is dynamically adjusted. By utilizing a theft mechanism and preset task allocation strategy, the task volume of the task queue is controlled to ensure that memory usage is within a controllable range. Combined with metadata and routing planning, task allocation and consumer execution are optimized.
It achieves efficient and stable data migration, avoids task backlog and resource congestion, improves the stability and overall efficiency of data migration, and prevents migration failure.
Smart Images
Figure CN122308727A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data migration method and related equipment based on queue water level. Background Technology
[0002] Existing parallel migration systems generally employ a loosely coupled model of "multiple producers in the producer layer and multiple consumers in the consumer layer." However, in scenarios with massive amounts of small files, the speed differences between components are significant. Due to the lack of closed-loop flow control based on queue levels, when the production rate far exceeds the consumption rate, tasks accumulate unbounded in memory, triggering frequent garbage collection (GC) or even out-of-memory (OOM) errors, leading to migration interruptions. When the consumer suddenly becomes idle, it cannot promptly wake up production, resulting in a "pipeline stall" and a sharp drop in overall throughput. The task accumulation further triggers the spread of congestion on network and storage resources, and the retry storm exacerbates latency, creating a vicious cycle that ultimately causes migration failure.
[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is related technology. Summary of the Invention
[0004] The main purpose of this application is to provide a data migration method and related equipment based on queue water level, aiming to solve the technical problem of how to perform data migration efficiently and stably.
[0005] To achieve the above objectives, this application proposes a data migration method based on queue water level, which includes:
[0006] In response to a data migration command, the system continuously retrieves metadata corresponding to multiple sets of data to be migrated from the source cluster. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. Based on the aforementioned metadata and theft mechanism, multiple producers in the producer layer are invoked to generate multiple data migration tasks, and the task queue corresponding to each data migration task is determined, as well as the task allocation speed corresponding to each task queue is determined. Based on the water level corresponding to each task queue and the preset task allocation strategy, the task allocation speed is dynamically adjusted, and based on the adjusted task allocation speed, the data migration task is allocated to the corresponding task queue. Based on the data migration tasks and theft mechanism in each task queue, multiple consumers in the consumer layer are invoked to migrate the data to be migrated.
[0007] In one embodiment, the step of dynamically adjusting the task allocation speed based on the water level corresponding to each task queue and a preset task allocation strategy, and allocating the data migration task to the corresponding task queue based on the adjusted task allocation speed, further includes: Get the water level, preset high water level, and preset low water level for each task queue; Based on the water level, the preset high water level line, and the preset low water level line, determine the water level status corresponding to each task queue; Based on the water level status and the preset task allocation strategy, the task allocation speed is dynamically adjusted, and based on the adjusted task allocation speed, the data migration task is allocated to the corresponding task queue.
[0008] In one embodiment, the water level status includes a high water level status, a low water level status, and a normal water level status. The step of dynamically adjusting the task allocation speed based on the water level status and a preset task allocation strategy, and allocating the data migration task to the corresponding task queue based on the adjusted task allocation speed, further includes: If the water level is high, based on the high water level and the preset task allocation strategy, the task allocation speed is reduced, and based on the reduced task allocation speed, the data migration task is allocated to the corresponding task queue. If the water level is normal, the data migration task is assigned to the corresponding task queue based on the normal water level, the preset task allocation strategy, and the task allocation speed. If the water level is low, based on the low water level and the preset task allocation strategy, the task allocation speed is increased, and based on the increased task allocation speed, the data migration task is allocated to the corresponding task queue.
[0009] In one embodiment, the step of continuously obtaining metadata corresponding to multiple sets of data to be migrated from the source cluster in response to a data migration instruction further includes: In response to a data migration command, determine the performance of the server corresponding to the metadata target in the source cluster; Based on the aforementioned performance, the metadata access speed is dynamically adjusted; Based on the adjusted metadata access speed, metadata corresponding to multiple sets of data to be migrated is continuously obtained from the source cluster.
[0010] In one embodiment, if there are idle producers / consumers in the producer / consumer layer, the stealing mechanism is used to call the idle producers / consumers to steal tasks from the task queues corresponding to the busy producers / consumers, and then call the idle producers / consumers to process the stolen tasks.
[0011] In one embodiment, the step of determining the task queue corresponding to each data migration task further includes: Based on the metadata, determine the physical location identifier corresponding to each group of data to be migrated, wherein the physical location identifier includes the object storage target; Based on the physical location identifier, determine the route corresponding to each group of data to be migrated; Based on the routing, the task queue corresponding to the data migration task for each group of data to be migrated is determined.
[0012] Furthermore, to achieve the above objectives, this application also proposes a data migration device based on queue level, the data migration device based on queue level comprising: The acquisition module is used to continuously acquire metadata corresponding to multiple sets of data to be migrated from the source cluster in response to the data migration command. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. The calling module is used to call multiple producers in the producer layer to generate multiple data migration tasks based on the metadata and theft mechanism, and to determine the task queue corresponding to each data migration task and the task allocation speed corresponding to each task queue. An adjustment module is used to dynamically adjust the task allocation speed based on the water level corresponding to each task queue and a preset task allocation strategy, and to allocate the data migration task to the corresponding task queue based on the adjusted task allocation speed. The migration module is used to call multiple consumers in the consumer layer to migrate the data to be migrated based on the data migration tasks and theft mechanism in each task queue.
[0013] In one embodiment, the adjustment module further includes: The first acquisition unit is used to acquire the water level, preset high water level line and preset low water level line corresponding to each task queue; The first determining unit is used to determine the water level status corresponding to each task queue based on the water level, the preset high water level line, and the preset low water level line. The first adjustment unit is used to dynamically adjust the task allocation speed based on the water level status and the preset task allocation strategy, and to allocate the data migration task to the corresponding task queue based on the adjusted task allocation speed.
[0014] In one embodiment, the water level status includes a high water level status, a low water level status, and a normal water level status, and the adjustment module further includes: The second adjustment unit is used to reduce the task allocation speed based on the high water level state and the preset task allocation strategy if the water level state is a high water level state, and to allocate the data migration task to the corresponding task queue based on the reduced task allocation speed. The third adjustment unit is used to allocate the data migration task to the corresponding task queue based on the normal water level state, the preset task allocation strategy and the task allocation speed if the water level state is normal. The fourth adjustment unit is used to, if the water level is low, increase the task allocation speed based on the low water level and a preset task allocation strategy, and allocate the data migration task to the corresponding task queue based on the increased task allocation speed.
[0015] In one embodiment, the acquisition module further includes: The second determining unit is used to determine the performance of the server corresponding to the metadata target in the source cluster in response to the data migration instruction; The fifth adjustment unit is used to dynamically adjust the metadata access speed based on the aforementioned performance. The second acquisition unit is used to continuously acquire metadata corresponding to multiple sets of data to be migrated from the source cluster based on the adjusted metadata access speed.
[0016] In one embodiment, the data migration device based on queue level is further configured to: If there are idle producers / consumers in the producer / consumer layer, the stealing mechanism is used to call the idle producers / consumers to steal tasks from the task queues corresponding to the busy producers / consumers, and then call the idle producers / consumers to process the stolen tasks.
[0017] In one embodiment, the calling module further includes: The third determining unit is used to determine the physical location identifier corresponding to each group of data to be migrated based on the metadata, wherein the physical location identifier includes the object storage target; The fourth determining unit is used to determine the route corresponding to each group of data to be migrated based on the physical location identifier; The fifth determining unit is used to determine the task queue corresponding to the data migration task for each group of data to be migrated, based on the route.
[0018] In addition, to achieve the above objectives, this application also proposes a data migration device based on queue level, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data migration method based on queue level as described above.
[0019] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the data migration method based on queue level as described above.
[0020] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the data migration method based on queue level as described above.
[0021] One or more technical solutions proposed in this application have at least the following technical effects: This application proposes a data migration method and related equipment based on queue levels, relating to the field of data processing technology. In contrast to existing parallel migration systems that generally employ a loosely coupled model of "multiple producers in the producer layer and multiple consumers in the consumer layer," in scenarios with massive amounts of small files, the speed differences between components are significant. Due to the lack of closed-loop flow control based on queue levels, when the production rate far exceeds the consumption rate, tasks accumulate unbounded in memory, triggering frequent GC or even OOM, leading to migration interruption. When the consumer suddenly becomes idle, it cannot wake up production in time, resulting in "pipeline stall," a sharp drop in overall throughput, and task accumulation further causing network and storage resource congestion, retry storms exacerbating latency, forming a vicious cycle, and ultimately causing migration failure. In this application, firstly, in response to data migration instructions, a continuous flow of data from the source cluster is used... The process involves acquiring metadata corresponding to multiple sets of data to be migrated. The source cluster includes a metadata target and multiple object storage targets. The metadata target stores metadata, and the object storage targets store the data to be migrated. Then, based on the metadata and a theft mechanism, multiple producers in the producer layer are invoked to generate multiple data migration tasks. The task queue corresponding to each data migration task is determined, as well as the task allocation speed corresponding to each task queue. Further, based on the water level corresponding to each task queue and a preset task allocation strategy, the task allocation speed is dynamically adjusted. Based on the adjusted task allocation speed, the data migration tasks are allocated to the corresponding task queues. Finally, based on the data migration tasks in each task queue and the theft mechanism, multiple consumers in the consumer layer are invoked to migrate the data to be migrated.
[0022] Understandably, this application dynamically adjusts the allocation speed of data migration tasks based on the water level of the task queue, thereby controlling the amount of tasks in each task queue, ensuring that the memory usage of each task queue is within a controllable range, and thus improving the stability of data migration and achieving efficient and stable data migration. Attached Figure Description
[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0024] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 This is a flowchart illustrating an embodiment of the data migration method based on queue water level in this application. Figure 2 A simplified flowchart illustrating the data migration method based on queue water level provided in Embodiment 1 of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the data migration method based on queue water level in this application; Figure 4 This is a schematic diagram of the module structure of the data migration device based on queue water level according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware operating environment involved in the data migration method based on queue water level in the embodiments of this application.
[0026] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0027] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0028] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0029] The main solution in this application embodiment is: In this embodiment, for ease of description, the following description will focus on a data migration device based on queue water level as the execution subject.
[0030] In related technologies, existing parallel migration systems generally adopt a loosely coupled model of "multiple producers in the producer layer and multiple consumers in the consumer layer." However, in scenarios with massive amounts of small files, the speed differences between components are significant. Due to the lack of closed-loop flow control based on queue levels, when the production rate far exceeds the consumption rate, tasks accumulate unbounded in memory, triggering frequent GC or even OOM (Out of Memory) errors, leading to migration interruptions. When the consumer suddenly becomes idle, it cannot wake up production in time, resulting in "pipeline stall" and a sharp drop in overall throughput. Task accumulation further triggers the spread of congestion on network and storage resources, and retry storms exacerbate latency, forming a vicious cycle that ultimately causes migration failure.
[0031] This application provides a solution in which: First, in response to a data migration instruction, metadata corresponding to multiple sets of data to be migrated is continuously obtained from a source cluster. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. Then, based on the metadata and a theft mechanism, multiple producers in the producer layer are invoked to generate multiple data migration tasks, and the task queue corresponding to each data migration task is determined, as well as the task allocation speed corresponding to each task queue is determined. Further, based on the water level corresponding to each task queue and a preset task allocation strategy, the task allocation speed is dynamically adjusted. Based on the adjusted task allocation speed, the data migration tasks are allocated to the corresponding task queues. Finally, based on the data migration tasks in each task queue and the theft mechanism, multiple consumers in the consumer layer are invoked to migrate the data to be migrated.
[0032] Understandably, this application dynamically adjusts the allocation speed of data migration tasks based on the task queue level, thereby controlling the number of tasks in each task queue, ensuring that the memory usage of each task queue is within a controllable range, and thus improving the stability of data migration and achieving efficient and stable data migration. It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a queue-level-based data migration device. The following description uses a queue-level-based data migration device as an example to illustrate this embodiment and the subsequent embodiments.
[0033] Based on this, embodiments of this application provide a data migration method based on queue water level, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the data migration method based on queue water level in this application.
[0034] In this embodiment, the data migration method based on queue water level includes steps S10~S40: Step S10: In response to the data migration instruction, continuously obtain metadata corresponding to multiple sets of data to be migrated from the source cluster. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. It should be noted that the migration is not scheduled or random, but is triggered by a migration command explicitly issued externally (operations platform, administrator script, automatic policy). The data migration command includes at least: source cluster address, target cluster address, and migration rules (which buckets / directories / prefixes, filtering conditions, concurrency, QoS, etc.).
[0035] It should be noted that in this application, "continuous" means using a batch + incremental polling approach, rather than fetching the entire data at once. Batch refers to fetching a fixed number of records per round (e.g., 1000 records) to avoid excessively large single RPC calls. Incremental refers to using the cursor / continuation-token or timestamp + sequence number of the source cluster to advance the data and achieve breakpoint resumption.
[0036] It should be noted that in this application, the data to be migrated is logically divided into non-overlapping shards, with each shard corresponding to a parallel task.
[0037] It should be noted that the metadata target only stores "index information", such as: object name, version number, size, etag, custom tag, ACL, and storage location pointer. It is usually composed of a small-capacity SSD with high IOPS and supports fast list and update.
[0038] It should be noted that object storage targets (OSTs) only store "physical data," such as shards / replicas / EC blocks, which consist of large-capacity HDDs / QLCs and provide high bandwidth to support data transmission.
[0039] In this application, a list is executed on the "metadata target" to obtain a list of objects and the necessary information related to each object. Furthermore, each piece of metadata is pushed into a memory queue (i.e., the "queue level" in the patent name) for consumption by the downstream data pipeline.
[0040] For example, to aid in understanding this embodiment, please refer to Figure 2 , Figure 2 A simplified flowchart of a data migration method based on queue water level is provided.
[0041] Understandably, after the migration command is issued, the system first continuously, in batches and in parallel retrieves object indexes from the "metadata pool" of the source cluster. By utilizing the architecture of physically separating metadata and entity data, it achieves parallel migration that is fast in listing, wide in reading, high in concurrency, and can be resumed from breakpoints.
[0042] Specifically, the step of continuously obtaining metadata corresponding to multiple sets of data to be migrated from the source cluster in response to the data migration command further includes steps S11 to S13: Step S11: In response to the data migration instruction, determine the performance of the server corresponding to the metadata target in the source cluster; It's important to note that before the migration begins, the system needs to perform a performance evaluation on the servers storing metadata in the source cluster. The performance of the metadata server directly affects the speed of metadata access, and thus the efficiency of the entire migration task. The performance evaluation includes the following aspects: CPU utilization: Assess whether the server's computing power is sufficient.
[0043] Memory usage: Check if there is enough memory to handle metadata requests.
[0044] I / O performance: Evaluate disk read and write speeds, especially for reading metadata.
[0045] Network bandwidth: Check whether the network connection between the server and the client is stable and efficient.
[0046] Current load: Assess the number of requests the server is currently processing to determine if it is overloaded.
[0047] Understandably, through these assessments, the system can obtain the real-time performance status of the metadata server, providing a basis for subsequent dynamic adjustments.
[0048] Step S12: Based on the performance, dynamically adjust the metadata access speed; It should be noted that in this embodiment, the system dynamically adjusts the metadata access speed based on the metadata server performance information obtained in step S11. The purpose of this adjustment is to balance the server load and the efficiency of the migration task. Specific adjustment strategies include: if the metadata server performance is poor (e.g., high CPU utilization, low I / O bandwidth, or high load), the system will reduce the metadata access speed to avoid putting excessive pressure on the server; if the metadata server performance is good, the system can appropriately increase the metadata access speed to accelerate the preparation phase of the migration task.
[0049] It should also be noted that, in addition to adjusting the speed of a single access, the system can also dynamically adjust the number of concurrent accesses. For example, if server performance allows, the number of concurrent requests can be increased to improve the efficiency of metadata retrieval.
[0050] Understandably, this dynamic adjustment mechanism ensures that the metadata server maintains good performance throughout the migration process, avoiding performance degradation or service unavailability due to overload.
[0051] Step S13: Based on the adjusted metadata access speed, continuously obtain metadata corresponding to multiple sets of data to be migrated from the source cluster.
[0052] In this embodiment, after adjusting the metadata access speed, the system begins to continuously retrieve metadata from the source cluster.
[0053] In this embodiment, the system retrieves metadata from the metadata server in batches based on the adjusted access speed. Each batch retrieves a certain amount of metadata to avoid overloading the server with too many requests at once. After retrieving one batch of metadata, the system continues to request the next batch as needed until all metadata for the data to be migrated is obtained.
[0054] It should be noted that metadata may be organized into multiple groups (e.g., by file type, directory structure, or data block), and the system will retrieve the metadata of these groups one by one.
[0055] Understandably, this method allows the system to efficiently retrieve metadata while avoiding excessive pressure on the metadata server. Furthermore, dynamically adjusting the access speed ensures stable metadata retrieval efficiency under varying server performance conditions.
[0056] Step S20: Based on the metadata and theft mechanism, call multiple producers in the producer layer to generate multiple data migration tasks, determine the task queue corresponding to each data migration task, and determine the task allocation speed corresponding to each task queue. It's important to note that metadata is crucial input here. Metadata contains detailed information about the data to be migrated, such as its size and storage location (e.g., its specific location within the object storage target). This information is used to assess the task's complexity and resource requirements, thus providing a basis for task generation and allocation.
[0057] Specifically, if there are idle producers / consumers in the producer / consumer layer, the stealing mechanism is used to call the idle producers / consumers to steal tasks from the task queues corresponding to the busy producers / consumers, and then call the idle producers / consumers to process the stolen tasks.
[0058] It's important to note that the task-stealing mechanism is a parallel task scheduling strategy used to address the problem of uneven task distribution. Its core idea is that each task queue (or thread) maintains its own task queue. If a thread's task queue is empty (i.e., there are no tasks to do), it can "steal" tasks from the task queues of other threads.
[0059] Understandably, the theft mechanism prevents some threads from becoming overloaded while others remain idle. At the same time, it dynamically allocates tasks based on the status of the real-time task queue, thereby improving the overall efficiency of the system.
[0060] It's important to note that in this system, the producer layer is responsible for generating data migration tasks. The producer's role is to generate specific migration tasks based on metadata. Each producer can generate tasks independently, thus achieving parallel task generation. For example, assuming there are 1000 files in the metadata that need to be migrated, the producer can divide these files into multiple tasks (e.g., each task contains 10 files) based on information such as file size and priority. The producer layer can contain multiple producers, each responsible for generating a portion of the tasks, thereby accelerating task generation.
[0061] It should be noted that each generated task needs to be assigned to a task queue. The purpose of the task queue is to temporarily store tasks to be processed and to allow consumers (threads that execute tasks) to retrieve tasks from it.
[0062] It should be noted that task queue allocation can be based on various strategies, such as: assigning high-priority tasks to high-priority queues; allocating tasks according to the current queue load to avoid overloading any queue; and assigning related tasks to the same queue to reduce data access latency. In this application, task allocation primarily relies on user-predefined OSTs.
[0063] It should be noted that task allocation speed refers to the rate at which tasks are assigned to the queue. Determining this speed requires consideration of the following factors: Current load of the queue: If the queue is already full, the allocation speed can be slowed down appropriately.
[0064] Consumer processing power: If the consumer (the thread executing the task) has strong processing power, the allocation speed can be accelerated.
[0065] Overall system resource utilization: Avoid over-allocation that could lead to system resource depletion.
[0066] Understandably, in this application, by utilizing metadata to assess task complexity and combining it with a theft mechanism to dynamically adjust task allocation, the system can achieve efficient task scheduling and resource utilization.
[0067] Specifically, the step of determining the task queue corresponding to each data migration task further includes steps S21 to S23: Step S21: Based on the metadata, determine the physical location identifier corresponding to each group of data to be migrated, wherein the physical location identifier includes the object storage target; It's important to note that in this step, the system needs to extract the physical storage location of each group of data to be migrated from the metadata. The physical location identifier refers to the actual storage location of the data in the source cluster, including the following: Object Storage Target: The specific location where data is stored, such as the bucket name, directory path, file name, etc.
[0068] Storage Node: The physical server or storage device where the data resides.
[0069] Data Block Location: In a distributed storage system, data may be divided into multiple blocks and stored on different nodes. Physical location identification needs to specify the exact location of these data blocks.
[0070] Understandably, by determining the physical location identifier, the system can accurately locate the specific storage location of each group of data in the source cluster, providing basic information for subsequent data reading and migration operations.
[0071] Step S22: Based on the physical location identifier, determine the route corresponding to each group of data to be migrated; It should be noted that after determining the physical location of the data, the system needs to further plan how to read this data from the source cluster. This involves data access routing planning.
[0072] It should be noted that routing refers to the transmission path from the data storage location to the migration system. This step needs to consider the following: Network topology of storage nodes: Determine the optimal network path from the storage nodes to the migration system. For example, choose the path with the highest bandwidth and lowest latency.
[0073] Load balancing: Select the optimal routing path based on the current network status and the load of storage nodes to avoid network congestion.
[0074] Data locality: If data is stored on multiple nodes, try to select the node closest to the migration system to reduce data transfer overhead.
[0075] Understandably, by determining the route, the system can efficiently read data from the source cluster, ensuring network transmission efficiency and stability during data migration.
[0076] Step S23: Based on the routing, determine the task queue corresponding to the data migration task for each group of data to be migrated.
[0077] In this embodiment, after determining the data transmission path, the system needs to allocate this data to specific task queues. The task queue is the execution unit of the migration task, responsible for allocating data migration tasks to consumers (task execution threads).
[0078] Understandably, through these three steps, the system can efficiently and accurately execute data migration tasks, ensuring that data is successfully migrated from the source cluster to the target cluster. This process embodies the core idea of "metadata-driven task scheduling" in the data migration system and is a key link in achieving efficient data migration.
[0079] Step S30: Based on the water level corresponding to each task queue and the preset task allocation strategy, dynamically adjust the task allocation speed, and allocate the data migration task to the corresponding task queue based on the adjusted task allocation speed. It's important to note that in a task scheduling system, the "water level" is a crucial monitoring metric used to measure the current load status of the task queue. The water level is typically a numerical value representing the number of tasks in the queue or the total size of the tasks (e.g., in bytes). A high water level reflects the queue's load: High water level: Many tasks in the queue, heavy load. Low water level: Few tasks in the queue, light load. Monitoring and adjusting the water level is key to dynamic task allocation, as it directly affects the speed and direction of task assignment.
[0080] It should be noted that the preset task allocation strategy is a predefined task allocation rule used by the system to guide how tasks are assigned to different queues. Preset strategies typically include the following common methods: Load balancing strategy: Dynamically allocate tasks based on the current load (water level) of the queue, giving priority to assigning tasks to queues with lighter loads.
[0081] Priority strategy: Tasks are assigned based on their priority, with higher priority tasks being assigned to less loaded queues.
[0082] Locality strategy: Assign related tasks to the same queue to reduce data access latency and improve cache utilization.
[0083] Resource utilization strategy: Dynamically adjust task allocation speed based on the overall resource utilization of the system to avoid resource overload.
[0084] It should be noted that task allocation speed refers to the rate at which tasks are assigned to the queue. The purpose of dynamically adjusting the task allocation speed is to optimize the overall system performance and resource utilization. The adjustment logic is as follows: High water level: If the water level of a task queue is high, it means that the queue is heavily loaded. At this time, the speed of assigning tasks to the queue should be slowed down to avoid further increasing the load.
[0085] Low water level: If the water level of a task queue is low, it means that the queue is lightly loaded. At this time, the speed of assigning tasks to the queue should be accelerated to make full use of its idle resources.
[0086] In this application, in addition to the water level, adjustments are also made in conjunction with a preset task allocation strategy. For example, if a queue has a low load but is currently processing high-priority tasks, the allocation speed may be adjusted appropriately to ensure that high-priority tasks are processed first.
[0087] It should be noted that after adjusting the task allocation speed, the system allocates the generated data migration tasks to the corresponding task queues based on the current water level and preset strategies. The allocation process is as follows: First, select the most suitable task queue based on the water level and strategy. For example, choose a queue with a low water level that meets the task priority requirements. Alternatively, allocate queues according to a pre-defined OST, and then adjust the task allocation speed based on the water level and strategy. Further, place the tasks into the selected task queues for subsequent processing by the consumer (task execution thread).
[0088] Understandably, this mechanism enables flexibility and efficiency in task allocation, ensuring load balancing and maximizing resource utilization within the system.
[0089] Step S40: Based on the data migration tasks and theft mechanism in each task queue, call multiple consumers in the consumer layer to migrate the data to be migrated.
[0090] It's important to note that, as described earlier, tasks have been generated by the producer layer and allocated to different task queues. Each task queue contains a series of data tasks to be migrated; these tasks are the basic units of data migration. For example, a task might represent migrating a single file or a group of files. These task queues are the source of tasks for consumers (task execution threads). The management and scheduling of task queues are crucial for the efficient operation of the entire data migration system.
[0091] It's important to note that the consumer layer is the actual executor of the tasks, responsible for retrieving tasks from the task queue and performing data migration. The consumer layer typically contains multiple consumers (threads or processes), each responsible for processing the tasks assigned to it. The role of the consumer includes: Task Acquisition: Retrieves tasks from the task queue. If the local queue is empty, tasks can be acquired from other queues via a stealing mechanism.
[0092] Task execution: Perform data migration operations based on the task description (such as data source and target location).
[0093] Results feedback: The results of task execution (success or failure) are fed back to the system for subsequent processing or retry.
[0094] It is understood that in this application, through dynamic task allocation and theft mechanisms, the system can achieve load balancing, improve task execution efficiency, and ensure the efficiency and reliability of data migration.
[0095] This application proposes a data migration method and related equipment based on queue levels, relating to the field of data processing technology. In contrast to existing parallel migration systems that generally employ a loosely coupled model of "multiple producers in the producer layer and multiple consumers in the consumer layer," in scenarios with massive amounts of small files, the speed differences between components are significant. Due to the lack of closed-loop flow control based on queue levels, when the production rate far exceeds the consumption rate, tasks accumulate unbounded in memory, triggering frequent GC or even OOM, leading to migration interruption. When the consumer suddenly becomes idle, it cannot wake up production in time, resulting in "pipeline stall," a sharp drop in overall throughput, and task accumulation further causing network and storage resource congestion, retry storms exacerbating latency, forming a vicious cycle, and ultimately causing migration failure. In this application, firstly, in response to data migration instructions, a continuous flow of data from the source cluster is used... The process involves acquiring metadata corresponding to multiple sets of data to be migrated. The source cluster includes a metadata target and multiple object storage targets. The metadata target stores metadata, and the object storage targets store the data to be migrated. Then, based on the metadata and a theft mechanism, multiple producers in the producer layer are invoked to generate multiple data migration tasks. The task queue corresponding to each data migration task is determined, as well as the task allocation speed corresponding to each task queue. Further, based on the water level corresponding to each task queue and a preset task allocation strategy, the task allocation speed is dynamically adjusted. Based on the adjusted task allocation speed, the data migration tasks are allocated to the corresponding task queues. Finally, based on the data migration tasks in each task queue and the theft mechanism, multiple consumers in the consumer layer are invoked to migrate the data to be migrated.
[0096] Understandably, this application dynamically adjusts the allocation speed of data migration tasks based on the water level of the task queue, thereby controlling the amount of tasks in each task queue, ensuring that the memory usage of each task queue is within a controllable range, and thus improving the stability of data migration and achieving efficient and stable data migration.
[0097] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The step of dynamically adjusting the task allocation speed based on the water level corresponding to each task queue and the preset task allocation strategy, and allocating the data migration task to the corresponding task queue based on the adjusted task allocation speed, further includes steps A10 to A30: Step A10: Obtain the water level, preset high water level line, and preset low water level line corresponding to each task queue; It's important to note that in a task scheduling system, the water level is a key monitoring metric used to measure the current load on the task queue. The water level is typically a numerical value representing the number of tasks in the queue or the total size of the tasks (e.g., in bytes).
[0098] It should be noted that the preset high and low watermarks are user-defined thresholds used to determine the load status of the task queue. The specific definitions are as follows: Preset High Water Mark: When the water level in the queue exceeds this threshold, the queue is considered to be heavily loaded, and the task allocation speed needs to be slowed down.
[0099] Low Water Mark (LWM): When the water level in the queue is below this threshold, the queue is considered to be lightly loaded, and task allocation can be sped up.
[0100] Understandably, these two thresholds are important bases for dynamically adjusting the task allocation speed.
[0101] Step A20: Based on the water level, the preset high water level line, and the preset low water level line, determine the water level status corresponding to each task queue; It should be noted that after obtaining the current water level and preset high and low water levels of the task queue, the system needs to determine the water level status of each task queue. Water level status typically falls into the following categories: Normal state: The current water level is between the preset high water level line and the low water level line, indicating that the queue load is moderate.
[0102] High water level status: The current water level exceeds the preset high water level line, indicating that the queue is heavily loaded.
[0103] Low water level status: The current water level is lower than the preset low water level line, indicating that the queue load is relatively light.
[0104] Understandably, by judging the water level status, the system can decide how to adjust the task allocation speed to optimize the overall load balancing.
[0105] Step A30: Based on the water level status and the preset task allocation strategy, dynamically adjust the task allocation speed, and based on the adjusted task allocation speed, allocate the data migration task to the corresponding task queue.
[0106] It should be noted that in this embodiment, the system dynamically adjusts the task allocation speed based on the water level status of the task queue and the preset task allocation strategy.
[0107] Understandably, by monitoring the task queue's status and combining it with a preset task allocation strategy, the task allocation speed can be dynamically adjusted, and tasks can be assigned to appropriate task queues. This mechanism can effectively optimize task allocation efficiency and load balancing, ensuring the efficient execution of data migration tasks.
[0108] Specifically, the water level status includes high water level status, low water level status, and normal water level status. The step of dynamically adjusting the task allocation speed based on the water level status and a preset task allocation strategy, and allocating the data migration task to the corresponding task queue based on the adjusted task allocation speed, further includes steps A31 to A33: Step A31: If the water level is high, based on the high water level and the preset task allocation strategy, reduce the task allocation speed, and based on the reduced task allocation speed, allocate the data migration task to the corresponding task queue. It should be noted that when the water level in the task queue exceeds the preset high water level, the system determines that the queue is in a high water level state. This indicates that there are many tasks in the queue and the load is heavy. When the system recognizes that the queue is overloaded, it needs to take measures to alleviate the pressure on the queue.
[0109] In this application, the system reduces the task allocation speed based on specific parameters of a preset strategy (such as the reduction ratio or a specific value). For example, if the current allocation speed is 10 tasks per second, it may be adjusted to 5 tasks per second. The system then allocates data migration tasks to the task queue according to the adjusted speed.
[0110] It should be noted that although the allocation rate is reduced, tasks will continue to be allocated until the queue level drops to a normal range. This adjustment ensures that the queue does not overwhelm the task pool while allowing tasks to be completed gradually.
[0111] Step A32: If the water level is normal, the data migration task is assigned to the corresponding task queue based on the normal water level, the preset task allocation strategy, and the task allocation speed. It should be noted that when the task queue level is between the preset high and low levels, the system determines that the queue is in a normal state. This indicates that the queue load is moderate, and task allocation and processing are balanced. The system recognizes that the queue load is moderate and there is no need to significantly adjust the task allocation speed.
[0112] It should be noted that the preset strategy will maintain a reasonable task allocation rate based on the current load. For example, if the strategy is load balancing, the system will continue to distribute tasks evenly across the queues. In this state, the task allocation rate usually remains constant, or is fine-tuned based on other factors (such as task priority, resource utilization, etc.).
[0113] In this embodiment, the system allocates data migration tasks to the task queue at a preset speed. Task allocation remains stable to ensure the queue can process tasks efficiently while avoiding resource waste or overload.
[0114] Step A33: If the water level is low, based on the low water level and the preset task allocation strategy, increase the task allocation speed, and based on the increased task allocation speed, allocate the data migration task to the corresponding task queue.
[0115] It should be noted that when the water level in the task queue falls below the preset low water level, the system determines that the queue is in a low water level state. This indicates that there are relatively few tasks in the queue and the load is light. Recognizing the light queue load, the system can increase the task allocation speed to fully utilize the queue's idle resources.
[0116] It's important to note that the preset strategy adjusts the task allocation speed based on the queue's low load. For example, the strategy might increase the allocation speed to improve resource utilization. The system will increase the task allocation speed based on the specific parameters of the preset strategy (such as the increase percentage or specific value). For instance, if the current allocation speed is 5 tasks per second, it might be adjusted to 10 tasks per second. The system then allocates data migration tasks to the task queue at the adjusted speed. By increasing the task allocation speed, the system can quickly fill the queue, fully utilize its processing capacity, and thus improve overall migration efficiency.
[0117] Understandably, these three steps together constitute a dynamic task allocation mechanism based on water level status, ensuring that the task queue can efficiently handle data migration tasks under different load conditions. This mechanism achieves load balancing and resource optimization through real-time monitoring and dynamic adjustment, which is an important guarantee for the efficient operation of the data migration system.
[0118] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the data migration method based on queue water level in this application. Any simple modifications based on this technical concept are within the protection scope of this application.
[0119] This application also provides a data migration device based on queue water level; please refer to... Figure 4 The data migration device based on queue water level includes: The acquisition module 10 is used to continuously acquire metadata corresponding to multiple sets of data to be migrated from the source cluster in response to the data migration instruction. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. The calling module 20 is used to call multiple producers in the producer layer to generate multiple data migration tasks based on the metadata and theft mechanism, and to determine the task queue corresponding to each data migration task and the task allocation speed corresponding to each task queue. Adjustment module 30 is used to dynamically adjust the task allocation speed based on the water level corresponding to each task queue and the preset task allocation strategy, and to allocate the data migration task to the corresponding task queue based on the adjusted task allocation speed. Migration module 40 is used to call multiple consumers in the consumer layer to migrate the data to be migrated based on the data migration tasks and theft mechanism in each task queue.
[0120] In one embodiment, the adjustment module further includes: The first acquisition unit is used to acquire the water level, preset high water level line and preset low water level line corresponding to each task queue; The first determining unit is used to determine the water level status corresponding to each task queue based on the water level, the preset high water level line, and the preset low water level line. The first adjustment unit is used to dynamically adjust the task allocation speed based on the water level status and the preset task allocation strategy, and to allocate the data migration task to the corresponding task queue based on the adjusted task allocation speed.
[0121] In one embodiment, the water level status includes a high water level status, a low water level status, and a normal water level status, and the adjustment module further includes: The second adjustment unit is used to reduce the task allocation speed based on the high water level state and the preset task allocation strategy if the water level state is a high water level state, and to allocate the data migration task to the corresponding task queue based on the reduced task allocation speed. The third adjustment unit is used to allocate the data migration task to the corresponding task queue based on the normal water level state, the preset task allocation strategy and the task allocation speed if the water level state is normal. The fourth adjustment unit is used to, if the water level is low, increase the task allocation speed based on the low water level and a preset task allocation strategy, and allocate the data migration task to the corresponding task queue based on the increased task allocation speed.
[0122] In one embodiment, the acquisition module further includes: The second determining unit is used to determine the performance of the server corresponding to the metadata target in the source cluster in response to the data migration instruction; The fifth adjustment unit is used to dynamically adjust the metadata access speed based on the aforementioned performance. The second acquisition unit is used to continuously acquire metadata corresponding to multiple sets of data to be migrated from the source cluster based on the adjusted metadata access speed.
[0123] In one embodiment, the data migration device based on queue level is further configured to: If there are idle producers / consumers in the producer / consumer layer, the stealing mechanism is used to call the idle producers / consumers to steal tasks from the task queues corresponding to the busy producers / consumers, and then call the idle producers / consumers to process the stolen tasks.
[0124] In one embodiment, the calling module further includes: The third determining unit is used to determine the physical location identifier corresponding to each group of data to be migrated based on the metadata, wherein the physical location identifier includes the object storage target; The fourth determining unit is used to determine the route corresponding to each group of data to be migrated based on the physical location identifier; The fifth determining unit is used to determine the task queue corresponding to the data migration task for each group of data to be migrated, based on the route.
[0125] The data migration device based on queue level provided in this application, employing the data migration method based on queue level in the above embodiments, can solve the technical problem of data migration based on queue level. Compared with related technologies, the beneficial effects of the data migration device based on queue level provided in this application are the same as those of the data migration method based on queue level provided in the above embodiments, and other technical features in the data migration device based on queue level are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0126] This application provides a data migration device based on queue level. The data migration device based on queue level includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data migration method based on queue level in the above embodiment 1.
[0127] The following is for reference. Figure 5 This document illustrates a structural diagram of a data migration device based on queue level suitable for implementing embodiments of this application. The data migration device based on queue level in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The data migration device based on queue level shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0128] like Figure 5As shown, the queue level-based data migration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the queue level-based data migration device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the queue-level-based data migration device to communicate wirelessly or wiredly with other devices to exchange data. Although a queue-level-based data migration device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.
[0129] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0130] The data migration device based on queue water level provided in this application, employing the data migration method based on queue water level in the above embodiments, can solve technical problems such as OOM and pipeline stall caused by rate mismatch in the prior art. Compared with related technologies, the beneficial effects of the data migration device based on queue water level provided in this application are the same as those of the data migration method based on queue water level provided in the above embodiments, and other technical features of the data migration device based on queue water level are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0131] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0133] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the data migration method based on queue level in the above embodiments.
[0134] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0135] The aforementioned computer-readable storage medium may be included in a queue level-based data migration device; or it may exist independently and not assembled into a queue level-based data migration device.
[0136] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the queue level-based data migration device, cause the queue level-based data migration device to: In response to a data migration command, the system continuously retrieves metadata corresponding to multiple sets of data to be migrated from the source cluster. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. Based on the aforementioned metadata and theft mechanism, multiple producers in the producer layer are invoked to generate multiple data migration tasks, and the task queue corresponding to each data migration task is determined, as well as the task allocation speed corresponding to each task queue is determined. Based on the water level corresponding to each task queue and the preset task allocation strategy, the task allocation speed is dynamically adjusted, and based on the adjusted task allocation speed, the data migration task is allocated to the corresponding task queue. Based on the data migration tasks and theft mechanism in each task queue, multiple consumers in the consumer layer are invoked to migrate the data to be migrated.
[0137] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0139] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0140] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described data migration method based on queue level, thereby solving the technical problem of data migration based on queue level. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data migration method based on queue level provided in the above embodiments, and will not be repeated here.
[0141] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data migration method based on queue level as described above.
[0142] The computer program product provided in this application can solve the technical problem of data migration based on queue level. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the data migration method based on queue level provided in the above embodiments, and will not be repeated here.
[0143] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.
Claims
1. A method for data migration based on queue water level, characterized in that, The data migration method based on queue water level includes: In response to a data migration command, the system continuously retrieves metadata corresponding to multiple sets of data to be migrated from the source cluster. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. Based on the aforementioned metadata and theft mechanism, multiple producers in the producer layer are invoked to generate multiple data migration tasks, and the task queue corresponding to each data migration task is determined, as well as the task allocation speed corresponding to each task queue is determined. Based on the water level corresponding to each task queue and the preset task allocation strategy, the task allocation speed is dynamically adjusted, and based on the adjusted task allocation speed, the data migration task is allocated to the corresponding task queue. Based on the data migration tasks and theft mechanism in each task queue, multiple consumers in the consumer layer are invoked to migrate the data to be migrated.
2. The queue level based data migration method as claimed in claim 1, wherein, The step of dynamically adjusting the task allocation speed based on the water level corresponding to each task queue and a preset task allocation strategy, and allocating the data migration task to the corresponding task queue based on the adjusted task allocation speed, further includes: Get the water level, preset high water level, and preset low water level for each task queue; Based on the water level, the preset high water level line, and the preset low water level line, determine the water level status corresponding to each task queue; Based on the water level status and the preset task allocation strategy, the task allocation speed is dynamically adjusted, and based on the adjusted task allocation speed, the data migration task is allocated to the corresponding task queue.
3. The queue level based data migration method of claim 2, wherein, The water level status includes high water level, low water level, and normal water level. The step of dynamically adjusting the task allocation speed based on the water level status and a preset task allocation strategy, and allocating the data migration task to the corresponding task queue based on the adjusted task allocation speed, further includes: If the water level is high, based on the high water level and the preset task allocation strategy, the task allocation speed is reduced, and based on the reduced task allocation speed, the data migration task is allocated to the corresponding task queue. If the water level is normal, the data migration task is assigned to the corresponding task queue based on the normal water level, the preset task allocation strategy, and the task allocation speed. If the water level is low, based on the low water level and the preset task allocation strategy, the task allocation speed is increased, and based on the increased task allocation speed, the data migration task is allocated to the corresponding task queue.
4. The queue level based data migration method as claimed in claim 1, wherein, The step of continuously obtaining metadata corresponding to multiple sets of data to be migrated from the source cluster in response to the data migration command further includes: In response to a data migration command, determine the performance of the server corresponding to the metadata target in the source cluster; Based on the aforementioned performance, the metadata access speed is dynamically adjusted; Based on the adjusted metadata access speed, metadata corresponding to multiple sets of data to be migrated is continuously obtained from the source cluster.
5. The queue level based data migration method as claimed in claim 1, wherein, If there are idle producers / consumers in the producer / consumer layer, the stealing mechanism is used to call the idle producers / consumers to steal tasks from the task queues corresponding to the busy producers / consumers, and then call the idle producers / consumers to process the stolen tasks.
6. The queue level based data migration method as claimed in claim 1, wherein, The step of determining the task queue corresponding to each data migration task further includes: Based on the metadata, determine the physical location identifier corresponding to each group of data to be migrated, wherein the physical location identifier includes the object storage target; Based on the physical location identifier, determine the route corresponding to each group of data to be migrated; Based on the routing, the task queue corresponding to the data migration task for each group of data to be migrated is determined.
7. A data migration device based on queue water level, characterized in that, The data migration device based on queue water level includes: The acquisition module is used to continuously acquire metadata corresponding to multiple sets of data to be migrated from the source cluster in response to the data migration command. The source cluster includes a metadata target and multiple object storage targets. The metadata target is used to store metadata, and the object storage targets are used to store the data to be migrated. The calling module is used to call multiple producers in the producer layer to generate multiple data migration tasks based on the metadata and theft mechanism, and to determine the task queue corresponding to each data migration task and the task allocation speed corresponding to each task queue. An adjustment module is used to dynamically adjust the task allocation speed based on the water level corresponding to each task queue and a preset task allocation strategy, and to allocate the data migration task to the corresponding task queue based on the adjusted task allocation speed. The migration module is used to call multiple consumers in the consumer layer to migrate the data to be migrated based on the data migration tasks and theft mechanism in each task queue.
8. A data migration device based on queue water level, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the data migration method based on queue level as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the data migration method based on queue level as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the data migration method based on queue level as described in any one of claims 1 to 6.