Task processing method and apparatus, system, and related device

Abstract

A task processing method and apparatus, a system, and a related device, relating to the technical field of storage. The task processing apparatus receives a data migration task for migrating target data indicated by a source logical address to a destination logical address, the data migration task comprising the source logical address and the destination logical address; converting the source logical address into a first physical address, converting the destination logical address into a second physical address, and generating a migration request carrying the first physical address and the second physical address; and sending the migration request to a first storage device, so as to instruct the first storage device to migrate the target data from the first physical address to the second physical address. In this way, storage devices execute the process of transmitting data between different storage devices and processing same, thereby effectively reducing the number of IO stacks and network stacks needing to be executed, and improving task execution efficiency. Moreover, no CPU is required for intervention, such that the CPU can improve the performance of a storage system on the basis of more computing power.

Description

Task processing methods, devices, systems and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202410501154.2, filed on April 24, 2024, entitled “Task Processing Method, Apparatus, System and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of storage technology, and in particular to a task processing method, apparatus, system and related equipment. Background Technology

[0003] With the development of storage technology, storage media such as non-volatile memory express (NVMe) are widely used in storage systems. Storage systems typically involve east-west traffic; that is, the Central Processing Unit (CPU) in the storage system reads data from the storage medium and then writes the data (such as updated data) back to the storage medium. During this process, the CPU needs to frequently read the metadata corresponding to multiple data blocks on the storage medium and perform data reading and writing based on the read metadata. This causes the CPU to execute a significant amount of input / output (I / O) stack operations (for requesting metadata or data) and network stack operations (for transmitting data), thus consuming a large amount of CPU computing power and resulting in lower storage system performance.

[0004] Currently, the data migration task of east-west traffic is usually offloaded from the CPU to the heterogeneous processors of the storage system. For example, the CPU can instruct the data processing unit (DPU) in the storage system to handle the data migration task of east-west traffic.

[0005] However, while this method of offloading the data migration task to a heterogeneous processor frees up CPU computing power, the heterogeneous processor still needs to execute a significant amount of I / O stack and network stack to complete the data migration task. This results in the execution efficiency of the data migration task for east-west traffic remaining low. Summary of the Invention

[0006] This application provides a task processing method to improve the efficiency of handling data migration tasks with east-west traffic. Furthermore, this application also provides a corresponding task processing apparatus, storage system, computing device, computer-readable storage medium, and computer program product.

[0007] Firstly, this application provides a task processing method applied to a storage system. The storage system includes multiple storage devices, each of which may be, for example, a solid-state drive (SSD). These multiple storage devices are used for persistent data storage. The multiple storage devices may reside on the same storage server (or the same storage node), or they may reside on different storage servers. This method can be executed by a corresponding task processing device. Specifically, the task processing device receives a data migration task, which includes a source logical address and a destination logical address. The data migration task is used to migrate target data indicated by the source logical address to the destination logical address. The source logical address belongs to a first logical storage unit, and the destination logical address belongs to a second logical storage unit. The number of source logical addresses can be one or more, and the number of destination logical addresses can also be one or more. When there are multiple source logical addresses, these multiple source logical addresses may belong to the same first logical storage unit, or they may belong to multiple different first logical storage units. Similarly, when there are multiple destination logical addresses, these multiple destination logical addresses may belong to the same second logical storage unit, or they may belong to multiple different second logical storage units. The first logical storage unit corresponds to a first physical storage space in multiple storage devices, and the second logical storage unit corresponds to a second physical storage space in multiple storage devices. The first physical storage space and the second physical storage space can be physical storage spaces in the same storage device or physical storage spaces in different storage devices. After receiving a data migration task, the task processing device converts the source logical address into a first physical address in the first physical storage space, converts the destination logical address into a second physical address in the second physical storage space, and generates a migration request. This migration request carries both the first and second physical addresses, thereby converting a data migration task including logical addresses into a migration request including physical addresses. Finally, the task processing device sends the migration request to the first storage device in the multiple storage devices so that it can use the migration request to instruct the first storage device to migrate the target data from the first physical address to the second physical address.

[0008] Thus, during the data migration task, the task processing device can convert the data migration task into a migration request that includes a first physical address and a second physical address, which the storage device can recognize. In other words, it converts a data migration task with logical storage unit-level semantics into a task with storage device-level semantics that the storage device can understand. This allows the storage device to perform the process of transferring and processing data between different storage devices, thereby completing the data migration task. During this process, no intervention from the CPU or heterogeneous processors in the storage system is required. This effectively reduces the amount of I / O and network stacks required by the storage system, thereby improving the execution efficiency of the data migration task. Furthermore, since no CPU intervention is needed during the data migration task, CPU computing power is freed up, allowing the CPU to use more computing power to serve upper-layer user I / O, thus effectively improving the performance of the storage system.

[0009] In one possible implementation, the data migration task is specifically used to instruct the target data to be migrated from the source logical address to the destination logical address; or, the data migration task is specifically used to instruct the target data to be backed up to the destination logical address; or, the data migration task is used to instruct the reconstruction of the first data based on erasure coding (EC) technology according to the target data, and the storage of the first data to the second physical address.

[0010] In one possible implementation, the task processing device can further determine the service type corresponding to the data migration task; wherein, when the service type is a first type, the migration request is used to instruct the first storage device to migrate the target data from a first physical address to a second physical address; when the service type is a second type, the migration request is used to instruct the first storage device to back up the target data to the second physical address; when the service type is a third type, the migration request is used to instruct the first storage device to perform data reconstruction based on EC technology according to the target data at the first physical address. Thus, the task processing device generates different types of migration requests according to different data migration tasks to instruct the first storage device to perform different types of migration operations.

[0011] In one possible implementation, if the target data is located in the first storage device, the task processing device can also generate a data view corresponding to the first storage device based on the storage location of the target data on the first storage device. This data view describes the location distribution of the target data on the first storage device. Therefore, when generating a migration request, the task processing device can specifically generate the migration request based on the data view corresponding to the first storage device. In this way, the task processing device can plan and control the migration process of data indicated by different first physical addresses based on the data view, thereby optimizing the efficiency of migrating data from different physical storage spaces on the first storage device. This improves the execution efficiency of the data migration task, such as avoiding aggregated read traffic in EC backup scenarios or reducing the amount of I / O generated during data backup.

[0012] In one possible implementation, a data migration task is used to instruct the migration of target data from a source logical address to a destination logical address. The migration request generated by the task processing device is used to instruct the migration of the target data from a storage location indicated by a first physical address to a storage location indicated by a second physical address. In this case, the task processing device can use the migration request to control the first storage device to migrate data from one storage location to another. Correspondingly, after the data migration is completed, no data is stored at the storage location indicated by the first physical address.

[0013] In one possible implementation, the target data is stored in the storage system using EC (Extended Error Correction) technology. Specifically, the migration request generated by the task processing device instructs the first storage device to determine m data points from the target data, where m is an integer greater than 1; generate n checksums for the m data points based on EC technology, where n is a positive integer; and split the different data points within the stripe consisting of the m data points and n checksums into different storage devices for storage. Since the first storage device generates checksums locally for the m data points and stores the m data points and checksums separately on other storage devices, the newly generated checksums can be used to ensure the accuracy and reliability of the migrated target data in the storage system after migration. Furthermore, during the migration process, the first storage device does not need to read the target data from other storage devices, thus avoiding the accumulation of read traffic.

[0014] In one possible implementation, the data migration task instructs the target data to be backed up to a destination logical address. When the task processing device generates a migration request based on the data view corresponding to the first storage device, it may specifically determine, based on the data view corresponding to the first storage device, multiple pieces of second data in the target data that are to be migrated to the same physical address in the second physical storage space. Furthermore, for these multiple pieces of second data, a migration request is generated, instructing the multiple pieces of second data to be backed up in batches. Thus, for these multiple pieces of second data, the task processing device can control the first storage device to perform batch backups, thereby reducing the amount of I / O generated during the data migration process and improving data migration efficiency.

[0015] In one possible implementation, the target data is stored in the storage system based on EC (Extended Equipping) technology. A migration request instructs a first storage device to reconstruct the first data based on the target data and store the first data at a second physical address within the first storage device. This allows the first storage device to perform the process of retrieving data from other storage devices and reconstructing the data without intervention from the CPU or heterogeneous processors in the storage system. This effectively reduces the amount of I / O and network stacks required by the storage system, thereby improving the efficiency of data reconstruction. Furthermore, the CPU can utilize more computing power to serve upper-layer user I / O, thus effectively improving the performance of the storage system.

[0016] In one possible implementation, multiple storage devices interact with each other via an interconnect bus; alternatively, they interact via a network interface card (NIC), a smart NIC, or a data processing unit (DPU). This allows multiple storage devices to achieve peer-to-peer interconnection based on a bus, NIC, smart NIC, or DPU. Furthermore, some or all of the storage devices may be configured with hardware that enables them to perform corresponding processing operations based on received migration requests.

[0017] Secondly, this application provides a task processing apparatus applied to a storage system, the storage system including multiple storage devices for persistently storing data. The task processing apparatus includes: an acquisition module for receiving a data migration task, the data migration task including a source logical address and a destination logical address, the data migration task being used to migrate target data indicated by the source logical address to the destination logical address, the source logical address belonging to a first logical storage unit, the destination logical address belonging to a second logical storage unit, the first logical storage unit corresponding to a first physical storage space among the multiple storage devices, and the second logical storage unit corresponding to a second physical storage space among the multiple storage devices; a conversion module for converting the source logical address to a first physical address in the first physical storage space, and converting the destination logical address to a second physical address in the second physical storage space; a generation module for generating a migration request, the migration request carrying the first physical address and the second physical address; and a communication module for sending the migration request to the first storage device among the multiple storage devices, the migration request being used to instruct the first storage device to migrate the target data from the first physical address to the second physical address.

[0018] In one possible implementation, the data migration task is used to instruct the target data to be migrated from the source logical address to the destination logical address; or, the data migration task is used to instruct the target data to be backed up to the destination logical address; or, the data migration task is used to instruct the first data to be reconstructed based on the target data using EC (erasure coding) technology, and the first data to be stored in the second physical address.

[0019] In one possible implementation, the generation module is further configured to: determine the service type corresponding to the data migration task; wherein, when the service type is a first type, the migration request is used to instruct the first storage device to migrate the target data at the first physical address to the second physical address; when the service type is a second type, the migration request is used to instruct the first storage device to back up the target data to the second physical address; when the service type is a third type, the migration request is used to instruct the first storage device to perform data reconstruction based on EC technology according to the target data at the first physical address.

[0020] In one possible implementation, the target data is located in a first storage device. The generation module is further configured to generate a data view corresponding to the first storage device based on the storage location of the target data on the first storage device. The data view is used to describe the location distribution of the target data on the first storage device. Therefore, when generating a migration request, the generation module is specifically configured to generate a migration request based on the data view corresponding to the first storage device.

[0021] In one possible implementation, a data migration task is used to instruct the migration of target data from a source logical address to a destination logical address, and a migration request is used to instruct the migration of target data from a storage location indicated by a first physical address to a storage location indicated by a second physical address.

[0022] In one possible implementation, the target data is stored in the storage system based on EC technology; the migration request is specifically used to instruct: to determine m data from the target data, where m is an integer greater than 1; to generate n check data for the m data based on EC technology, where n is a positive integer; and to split the different data in the stripe consisting of the m data and the n check data into different storage devices for storage.

[0023] In one possible implementation, the data migration task is used to instruct the target data to be backed up to the destination logical address; then, when generating the migration request, the generation module is specifically used to: determine multiple second data in the target data that are to be migrated to the same physical address in the second physical storage space according to the data view corresponding to the first storage device; generate a migration request for the multiple second data, the migration request being used to instruct the multiple second data to be backed up in batch.

[0024] In one possible implementation, the target data is stored in the storage system based on EC technology, and the migration request is used to instruct the first storage device to reconstruct the first data according to the target data and store the first data at a second physical address in the first storage device.

[0025] In one possible implementation, multiple storage devices interact with each other via an interconnect bus; or, multiple storage devices interact with each other via a network interface card (NIC), a smart NIC, or a data processing unit (DPU).

[0026] Thirdly, this application provides a storage system including a task processing device and a plurality of storage devices for persistently storing data, the plurality of storage devices including a first storage device; wherein, the task processing device is used to execute a task processing method as described in the first aspect or any implementation thereof, and the first storage device is further used to execute a migration request to migrate target data from a first physical address to a second physical address.

[0027] Fourthly, this application provides a computing device, which includes a processor and a memory. The processor and the memory communicate with each other. The processor executes instructions stored in the memory to cause the computing device to perform a task processing method as described in the first aspect or any implementation thereof. It should be noted that the memory may be integrated into the processor or may be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0028] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operational steps of the task processing method described in the first aspect or any implementation thereof.

[0029] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the task processing method described in the first aspect or any implementation thereof.

[0030] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0031] Figure 1 is a schematic diagram of an exemplary storage system provided in this application;

[0032] Figure 2a is a schematic diagram of a heterogeneous processor performing a garbage collection task;

[0033] Figure 2b is a schematic diagram illustrating the reduction of the number of I / O operations and data transfers.

[0034] Figure 3 is a flowchart illustrating a task processing method provided in this application;

[0035] Figure 4 is a schematic diagram of various data migration tasks provided in this application;

[0036] Figure 5 is a schematic diagram of calculating the physical address based on the logical address;

[0037] Figure 6 is a schematic diagram of the generated data view provided in this application;

[0038] Figure 7 is a schematic diagram of generating migration requests and batch backup data based on data views in a replica scenario;

[0039] Figure 8 is a schematic diagram of generating migration requests and splitting striped data based on data views in the EC scenario;

[0040] Figure 9 is a schematic diagram of the structure of a task processing device provided in this application;

[0041] Figure 10 is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0042] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0043] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0044] Referring to Figure 1, which is a schematic diagram of an exemplary storage system 10. As shown in Figure 1, the storage system 10 includes multiple storage servers, each of which is configured with one or more storage devices. Figure 1 illustrates an example including two storage servers and N storage devices, namely, storage server 101, storage server 102, and storage devices 1 to N. Here, N is a positive integer greater than 1.

[0045] As shown in Figure 1, storage server 101 includes CPU 1011 and storage devices 1 to M, where M is a positive integer less than N. A task processing unit 200 can run on CPU 1011. The task processing unit 200 can be implemented in software; for example, it can be implemented using at least one of an engine, virtual machine, or component. In this case, the task processing unit 200 can also be called a disk move engine (DME). In practical applications, storage server 101 may also include other necessary components, such as memory and cache (not shown in Figure 1), which will not be elaborated further. Furthermore, CPU 1011 may also run a cache subsystem 201, an indexing subsystem 202, and a persistence subsystem 203, as shown in Figure 1. Each subsystem can be used to generate data migration tasks. The structure of storage server 102 is similar to that of storage server 101, as shown in Figure 1, and will not be elaborated further.

[0046] Storage devices 1 to N are used for persistent data storage. For example, each storage device can be a solid-state drive (SSD), a hard disk drive (HDD), or a hard disk. Furthermore, different storage devices can communicate with each other via wired or wireless means. For instance, different storage devices can exchange data based on an interconnect bus such as a universal bus (UB) or a compute express link (CXL) bus. Also, each storage device can be configured with a network interface card (NIC), a smart network interface card (SNIC), or a data processing unit (DPU), allowing different storage devices to exchange data via NICs, SNICs, or DPUs; this is not limited.

[0047] Furthermore, storage devices located on the same storage server can communicate through the internal bus of the storage server, while storage devices located on different storage servers can communicate through wired or wireless networks, i.e., cross-node communication.

[0048] Typically, the caching subsystem 201, indexing subsystem 202, and persistence subsystem 203 can all generate data migration tasks based on logical storage units at the business logic layer. A logical storage unit, also known as an object, refers to the basic unit in the storage system 10 that provides data storage for the business logic layer. Specifically, it abstracts one or more storage areas in the storage device into a logical entity capable of storing data for the business logic layer. For example, a data migration task may include the address of the data to be migrated in logical storage unit A, the data migration operation, and the address of the migrated data in logical storage unit B, indicating the migration of data from logical storage unit A to logical storage unit B. Since the data migration task includes relevant information about logical storage units, it belongs to the logical storage unit level semantics, or it can be called an object semantics task. Therefore, at the business logic layer, data reading, writing, and management can be performed on a logical storage unit basis. Correspondingly, in the underlying storage device, the storage system 10 will process the data within the physical address mapped to the logical storage unit according to the processing operation performed on that logical storage unit.

[0049] Each logical storage unit can have a unique identifier, such as a unique number assigned to the logical storage unit in storage system 10, and each logical storage unit corresponds to one or more storage regions on the storage device. In practical applications, different logical storage units can correspond to storage regions of the same size in the storage device. For example, a logical storage unit can be one of multiple Plogs included in a logical unit number (LUN), and each Plog can be used to store 64 megabits (MB) of data, mapping 64MB of storage space in the storage device, so that the 64MB of data can be stored on the storage device. Furthermore, each Plog in the LUN is uniquely identified.

[0050] For example, the data migration task generated by the caching subsystem 201 may be to persistently store data in the cache of the storage system 10. When data is backed up in the storage system 10, the data migration task generated by the caching subsystem 201 instructs the data in the cache to be stored in a storage device and the newly stored data in that storage device to be backed up to other storage devices, thereby generating east-west traffic.

[0051] The data migration tasks generated by the indexing subsystem 202, such as garbage collection tasks, are used to automatically identify and reclaim storage space occupied by objects not used by the program, thereby improving storage space utilization. During this process, the reclaimed valid data is first read from the storage device and then written back to the storage device, thus generating east-west traffic.

[0052] The data migration task generated by the persistent subsystem 203 may be, for example, a data migration task for a scenario of expanding storage capacity (i.e., capacity expansion), used to migrate data from some storage devices to the storage device used for capacity expansion, thereby generating east-west traffic.

[0053] For data migration tasks corresponding to east-west traffic, if the CPU in storage system 10 executes the data migration task, it will consume too much of the CPU's computing power. Consequently, much of the CPU's computing power cannot be used to serve user I / O in storage system 10, affecting the performance of storage system 10. While configuring heterogeneous processors (such as accelerator cards) in storage system 10 and having these heterogeneous processors execute the data migration task can free up CPU computing power, the heterogeneous processors still need to execute a significant amount of I / O stack and network layer operations, resulting in lower execution efficiency for the data migration task.

[0054] Taking a data migration task specifically as a garbage collection task as an example, as shown in Figure 2a, similar to the garbage collection task executed by the CPU, the heterogeneous processor sends an address request (executes an I / O stack) to the persistence layer (such as the persistence subsystem 203) based on the garbage collection task generated by the index layer. This address request includes the logical address in the logical storage unit corresponding to the valid data, used to request the physical address on the storage device mapped by the logical address (this physical address is used to persistently store the valid data on the storage device). The persistence layer records the mapping relationship between each logical address in the logical storage unit and the physical address on the storage device. Then, the heterogeneous processor accesses the storage device according to the obtained physical address (executes an I / O stack again) to obtain the valid data participating in garbage collection. During this process, each data block executes at least two I / O stack operations, and the valid data is transferred from the persistence layer to the index layer, involving one data transfer, i.e., at least one network stack operation. Next, the heterogeneous processor accesses the persistence layer in a similar manner, rewriting the reclaimed valid data back into the storage device. This storage device can be the original storage device (with valid data stored in another storage area within it) or a new storage device. This results in at least two more I / O stack operations and at least one more network stack operation, as shown in Figure 2b. Therefore, during the entire garbage collection process, the heterogeneous processor needs to execute at least four I / O stack operations and at least two network stack operations, as shown in Figure 2b. In practical applications, heterogeneous processors typically perform garbage collection on data from multiple data blocks. This results in a large number of I / O stack and network stack operations, leading to lower overall efficiency in garbage collection tasks.

[0055] Based on this, this application provides a task processing method aimed at improving the execution efficiency of data migration tasks for east-west traffic. Specifically, the task processing device 200 acquires a data migration task, which may be generated by, for example, a caching subsystem 201, an indexing subsystem 202, or a persistence subsystem 203. The acquired data migration task is a task with logical storage unit-level semantics, meaning it includes a source logical address in logical storage unit 1 and a destination logical address in logical storage unit 2, indicating the migration of the target data indicated by the source logical address to the destination logical address. Then, the task processing device 200 does not directly execute the data migration task, but instead converts it into a task that the storage device can understand, or what can be called a storage device-level semantic task (such as a disk-level semantic task). Specifically, the task processing device 200 converts the source logical address to a physical address 2 in the physical storage space 1 mapped by logical storage unit 1, and converts the destination logical address to a physical address 2 in the physical storage space 2 mapped by logical storage unit 2. Then, the task processing unit 200 generates a migration request carrying physical address 1 and physical address 2, and sends the migration request to the first storage device among the N storage devices included in the storage system 10. The first storage device can refer to one or more storage devices. At this time, since the addresses included in the migration request are all physical addresses on the storage devices, the first storage device can usually recognize and execute the migration request (that is, it can recognize the task with disk-level semantics). In this way, the first storage device can execute the migration request, migrate the target data at physical address 1 from physical address 1 to physical address 2, thereby realizing the migration of the target data indicated by the source logical address to the destination logical address.

[0056] Thus, during the data migration task in the east-west direction, the task processing device 200 can convert the data migration task into a migration request that the storage device can recognize, including physical address 1 (source physical address) and physical address 2 (destination physical address). In other words, it converts the data migration task with logical storage unit-level semantics into a task with storage device-level semantics that the storage device can understand. This allows the storage device in the storage system 10 to perform the process of transferring and processing data between different storage devices, thereby completing the data migration task. During this process, no intervention from the CPU or heterogeneous processors in the storage system 10 is required, which effectively reduces the number of I / O stacks and network stacks required by the storage system 10, thereby improving the execution efficiency of the data migration task.

[0057] Moreover, since no CPU intervention is required during the data migration process, the CPU's computing power can be freed up, allowing the CPU to serve upper-layer user I / O with more computing power, thereby effectively improving the performance of the storage system 10.

[0058] Taking the data migration task specifically as a garbage collection task as an example, the task processing device 200 can access the metadata in the persistent layer and, based on this metadata, convert the source logical address in the garbage collection task into the physical address 1 of the valid data to be reclaimed on storage device P, and convert the destination logical address in the garbage collection task into the physical address 2 of the reclaimed data to be written back to storage device Q. This converts the garbage collection task into a migration request including physical address 1 and physical address 2. Then, the task processing device 200 can send this migration request to storage device P. Storage device P can then recognize and execute the migration request, perform garbage collection on storage device P, and send the valid data retained after garbage collection to storage device Q via the bus or communication network between storage devices according to the indication information of physical address 2. Storage device Q then stores this valid data. During this process, for data in multiple data blocks on storage device P, the task processing device 200 only needs to execute the IO stack once (this IO stack is used to send the migration request and may not include the data in the data blocks), and can transfer data between storage devices once, as shown in Figure 2b. Furthermore, directly transferring data between storage devices can shorten data transmission paths, thereby improving the efficiency of garbage collection tasks.

[0059] It is worth noting that the storage system 10 shown in Figure 1 is merely an illustrative example and is not intended to be limiting. For example, in practical applications, the storage system 10 may include a greater number of storage servers. Alternatively, the storage server 101 may also include heterogeneous processors, allowing the task processing device 200 to be deployed as software within these heterogeneous processors. Alternatively, the task processing device 200 may be implemented using hardware deployed within the storage server. For instance, the task processing device 200 may be implemented using a processor, which may be any type of processor or any combination thereof, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). Furthermore, the number of processors included in the task processing device 200 can be arbitrary, and the types of processors included can be one or more. Specifically, the number and types of processors can be set according to the actual business needs of the application, and this application does not limit this.

[0060] For ease of understanding, embodiments of the task processing method provided in this application will be described below with reference to the accompanying drawings.

[0061] Referring to Figure 3, which is a flowchart illustrating a task processing method according to an embodiment of this application, this method can be applied to the storage system 10 shown in Figure 1, or to other applicable storage systems. For ease of explanation, this embodiment uses the storage system 10 shown in Figure 1 as an example for illustrative purposes.

[0062] The task processing method shown in Figure 3 may specifically include:

[0063] S301: The task processing device 200 receives a data migration task, which includes a source logical address and a destination logical address. The data migration task is used to instruct the target data indicated by the source logical address to be migrated to the destination logical address. The source logical address belongs to a first logical storage unit, and the destination logical address belongs to a second logical storage unit. The first logical storage unit corresponds to a first physical storage space in N storage devices, and the second logical storage unit corresponds to a second physical storage space in N storage devices.

[0064] In this embodiment, a data migration task refers to a task that generates east-west traffic, that is, a task that writes data from some storage devices to other storage devices based on data on some storage devices. For example, a data migration task may be used to instruct the migration or backup of data on some storage devices to other storage devices, or to instruct the processing of data on some storage devices and then writing the resulting new data to other storage devices.

[0065] In the first implementation example, when the tasks are divided according to their generation method, the data migration tasks can be tasks generated by different business logic layers, such as the caching subsystem 201, the indexing subsystem 202, or the persistence subsystem 203.

[0066] Data migration tasks can be tasks generated by the caching subsystem 201 to persistently store data in the cache to multiple storage devices (backup storage).

[0067] Alternatively, data migration tasks can be garbage collection tasks generated by the indexing subsystem 202, or tier down / up tasks. Among them, tier down / up tasks refer to migrating cold data to storage devices with lower read / write performance and hot data to storage devices with higher read / write performance, based on the data's popularity (e.g., determining the popularity of data based on the frequency of data access).

[0068] Alternatively, data migration tasks can be expansion, reduction (capacity reduction), reconstruction, or data ownership migration tasks generated by the persistence subsystem 203. Among these, reconstruction tasks refer to reconstructing a portion of data stored using EC technology from a portion of the data stored using EC technology, even if that portion was lost. For example, assuming a storage system uses six storage devices based on this EC mechanism to store data, if data on one storage device is lost, the data on the lost device can be reconstructed using data from the remaining storage devices, ensuring the reliability of data storage.

[0069] In the second implementation example, when divided according to the specifications of data movement, the data migration task can be a partial migration task, a full migration task, or a collaborative migration task. A partial migration task refers to moving a portion of the data located on multiple storage devices indicated by the first logical storage unit to other storage devices; as shown in Figure 4, a portion of the data block in a chunk (first logical storage unit) on disk 0 is migrated to disk 1. A full migration task refers to moving all the data located on multiple storage devices indicated by the first logical storage unit to other storage devices; as shown in Figure 4, all the data in a chunk on disk 0 is migrated to disk 1. A collaborative migration task refers to using the data located on the storage devices indicated by the first logical storage unit to reconstruct the data based on EC technology, and writing the reconstructed new data to other storage devices; as shown in Figure 4, data on disks 0 to 4 is reconstructed based on EC technology, and the reconstructed data is written to disk 5.

[0070] Typically, the data migration task acquired by the task processing device 200 may include a source logical address in the first logical storage unit and a destination logical address in the second logical storage unit, used to instruct the migration of the target data indicated by the source logical address to the destination logical address. During data migration between the source and destination logical addresses, various processing operations can be performed on the data. For example, the processing operation may be a data migration operation, instructing the migration of the data indicated by the source logical address to the destination logical address; in this case, after the data migration is completed, the source logical address no longer indicates data (the previously indicated target data is deleted). Alternatively, the processing operation may be a data backup operation, instructing the backup of the data indicated by the source logical address to the destination logical address; in this case, after the data migration is completed, the source logical address still indicates the target data. Another example is a data reconstruction operation, instructing the reconstruction of the data indicated by the source logical address and then writing the resulting data to the destination logical address. For example, the processing operation could specifically be an EC (erasure coding) calculation operation, which instructs the data indicated by the source logical address to undergo EC calculation to obtain the checksum, and then split the data and the checksum and write them to multiple different locations indicated by the destination logical address. Based on this, the migration request can also include processing operations, which can be data migration operations, data backup operations, data reconstruction operations, or EC calculation operations, or other types of operations, without limitation.

[0071] In one possible implementation, the task processing device 200 can provide different interfaces for data migration tasks of different business types, with each interface indicating a processing operation for a specific type of data. Then, after generating a data migration task, the caching subsystem 201, indexing subsystem 202, or persistence subsystem 203 can call the corresponding interface to provide the data migration task to the task processing device 200 based on the business type of the data migration task. In this way, the task processing device 200 can determine the business type of the data migration task based on the interface used when receiving it, and further determine the processing operations required for the data when migrating it between different storage devices based on the business type.

[0072] For example, when the data migration task is specifically a garbage collection task or a performance tier down / up task, the index subsystem 202 can call interface 1 to send the data migration task to the task processing device 200. Accordingly, the task processing device 200 can determine the business type corresponding to the data migration task as the first type (such as the business type of partial migration) according to interface 1, and determine to perform the migration of the target data indicated by the source logical address from one storage device to another.

[0073] For example, when the data migration task is specifically a persistent storage task, the caching subsystem 201 can call interface 2 to send the data migration task to the task processing device 200. Accordingly, the task processing device 200 can determine, based on interface 2, that the business type corresponding to the data migration task is the second type (such as the business type of overall migration), and determine to back up the target data indicated by the source logical address to multiple other storage devices.

[0074] For example, when the data migration task is specifically a reconstruction task, the persistent subsystem 203 can call interface 3 to send the data migration task to the task processing device 200. Accordingly, the task processing device 200 can determine, based on interface 3, that the business type corresponding to the data migration task is the third type (such as the collaborative migration business type), and determine to perform a data reconstruction operation on the target data indicated by the source logical address (the target data is scattered across multiple storage devices) based on EC technology.

[0075] The first and second logical storage units can be, for example, Plogs, chunks, or files. A chunk is a logical storage unit that maps one or more physical storage spaces on a storage device to a writable space in the storage system 10. When creating a logical storage unit, the storage system 10 abstracts one or more physical storage spaces on the storage device into a logical entity and assigns a unique identifier to this logical entity, such as a chunk number. The mapping relationship between logical storage units and physical storage spaces can be stored in the persistence layer. The processing operations performed can be, for example, data reconstruction operations or data migration operations, and are not limited thereto. For example, the source logical address can be indicated by the identifier of the first logical storage unit, the offset of the target data in the first logical storage unit, and the length of the target data. Similarly, the destination logical address can be indicated by the identifier of the second logical storage unit, the offset, and the data length; or, address 2 can only include the identifier of the second logical storage unit, such as allowing the user to specify that data is saved to the second logical storage unit, but without specifying the specific location of the data in the second logical storage unit.

[0076] In this embodiment, the first logical storage unit (and similarly the second logical storage unit) can refer to one logical storage unit or multiple logical storage units. For example, the first logical storage unit can refer to multiple chunks, etc., and there is no limitation on this.

[0077] S302: The task processing device 200 converts the source logical address into a first physical address in the first physical storage space and converts the destination logical address into a second physical address in the second physical storage space.

[0078] In this embodiment, the underlying storage devices typically have difficulty directly recognizing data migration tasks that include both source and destination logical addresses. Therefore, the task processing device 200 can first convert the data migration task into a migration request that the storage device can recognize, including the physical address, so that the storage device can subsequently execute the process of migrating data between different storage devices.

[0079] It is understood that since each logical storage unit in storage system 10 typically maps to different physical storage spaces on the storage device, performing processing operations on the target data indicated by the source logical address means performing processing operations on the data located in the storage device indicated by the physical address mapped to that source logical address. Therefore, task processing device 200 can translate each logical address in the data migration task into a corresponding physical address.

[0080] In one possible implementation, the persistent subsystem 203 in the storage system 10 can store metadata, which can be used to describe the mapping relationship between each logical storage unit and the physical storage space on the storage device. Thus, the task processing device 200 can access the persistent subsystem 203 to obtain the metadata corresponding to the first and second logical storage units, and then use this metadata to translate the logical addresses in the data migration task.

[0081] Specifically, as shown in Figure 1, the task processing device 200 may be configured with a unified data processing framework (UDPF). After obtaining a data migration task, the UDPF can access the persistence subsystem 203 to obtain the first metadata corresponding to the first logical storage unit and the second metadata corresponding to the second logical storage unit. The first metadata includes the starting address of the physical storage space in the storage device corresponding to the first logical storage unit, and the second metadata includes the starting address of the physical storage space in the storage device corresponding to the second logical storage unit. The physical storage spaces mapped to the first and second logical storage units may be located on different storage devices. For example, the source logical address may include the identifier of the first logical storage unit, the offset of the target data in the first logical storage unit, and the data length, etc. Then, the task processing device 200 accesses the persistence subsystem 203 based on the identifier of the first logical storage unit to obtain the first metadata corresponding to the first logical storage unit. Similarly, the task processing device 200 can obtain the second metadata corresponding to the second logical storage unit based on the identifier of the second logical storage unit included in address 2.

[0082] Then, UDPF can calculate the first physical address corresponding to the target data in the storage device, i.e., the first physical address mapped by the source logical address, based on the first metadata and the source logical address. For example, UDPF can calculate the starting storage location of the target data in the storage device based on the starting address of the physical storage space in the storage device corresponding to the first logical storage unit and the offset included in the source logical address. For example, the starting storage location can be identified by a logical block address (LBA). Next, UDPF can determine the physical storage space of the target data in the storage device based on the data length included in the source logical address and the starting storage location. Then, the address used to indicate the physical storage space is the first physical address of the target data in the storage device, such as the first physical address which may include the LBA and the data length.

[0083] For example, as shown in Figure 5, the first logical storage unit is specifically a chunk, and the target data is a portion of the data in this chunk. Then, the source logical address may include a chunk identifier, an offset of 1MB (megabytes), and a data length of 4KB (kilobytes). The starting address of the physical storage space in the storage device corresponding to the first logical storage unit is 100MB. Then, the task processing device 200 calculates the starting position of the target data stored on the storage device as 101MB based on the offset in address 1 and the starting address. Thus, based on the data length, the first physical address of the target data on the storage device can be determined to be from 101MB to (101MB + 4KB), as shown in Figure 5.

[0084] Furthermore, the task processing device 200 can also calculate the second physical address in the storage device mapped to the destination logical address based on the second metadata and the destination logical address in the second logical storage unit. It should be noted that no processing operation has been performed on the target data at this point; therefore, the calculated second physical address can be the physical address of the data generated by the subsequent processing operation on the target data, stored in the storage device.

[0085] S303: The task processing device 200 generates a migration request, which carries a first physical address and a second physical address.

[0086] In this embodiment, after the task processing device 200 converts the source logical address into a first physical address and the destination logical address into a second physical address, it can generate a migration request that includes the first physical address and the second physical address. This realizes the conversion of a data migration task with logical storage unit level semantics into a disk level semantic task (i.e., a migration request) that the storage device can recognize, so that the storage device can subsequently execute the process of migrating the target data indicated by the source logical address to the destination logical address.

[0087] In practical applications, since the storage system 10 includes multiple storage devices, the first physical address and the second physical address determined by the task processing device 200 can also be associated with the identifier of the storage device, so as to determine the storage device to which the first physical address belongs and the storage device to which the second physical address belongs using the associated identifier.

[0088] Furthermore, the migration request may also carry processing operations (such as including opcodes in the migration request) so that the processing operations can be used to instruct the storage device to process the target data.

[0089] S304: The task processing unit 200 sends the migration request to the first storage device among the N storage devices.

[0090] S305: The first storage device executes the migration request to migrate the target data from the first physical address to the second physical address.

[0091] In this context, the first storage device can be one or more storage devices from a pool of N storage devices. In a data migration scenario, the first storage device can be the storage device belonging to the physical storage space indicated by the first physical address. In an EC (Extended Storage Capture) reconstruction scenario, the first storage device can be the storage device belonging to the physical storage space indicated by the second physical address.

[0092] During the execution of a migration request, the first storage device can access the target data on the corresponding storage device based on the first physical address in the migration request, perform corresponding processing operations on the target data (as per the migration request), and then write the data obtained from the processing operations to the storage location indicated by the second physical address, based on the second physical address in the migration request. In practical applications, the first storage device can be configured with hardware for parsing the task, and can use this hardware to parse information such as the first physical address, processing operations, and the second physical address from the migration request, and perform processing operations on the target data indicated by the first physical address. Alternatively, the first storage device can be configured with software for parsing the task, and can use this software to parse information such as the first physical address, processing operations, and the second physical address from the migration request, and perform processing operations on the target data indicated by the first physical address.

[0093] Specifically, when the physical storage space indicated by the first physical address and the physical storage space indicated by the second physical address are different physical storage spaces in the same storage device, the first storage device can perform data migration internally, avoiding data transmission across storage devices in the communication network, which can effectively improve the efficiency of performing data migration tasks.

[0094] When the physical storage space indicated by the first physical address and the physical storage space indicated by the second physical address are physical storage spaces in different storage devices, the first storage device can transfer the data obtained by performing processing operations on the target data from the first storage device to the storage device to which the physical storage space indicated by the second physical address belongs, or read the target data from other storage devices into the first storage device to perform processing operations and store the obtained data in the first storage device. This can effectively shorten the data transmission path when performing data migration tasks (without having to transfer the data from the storage device to the CPU / heterogeneous processor and then back to the storage device), thereby improving the efficiency of performing data migration tasks.

[0095] In practical applications, the migration request may include one or more second physical addresses. When there is only one second physical address, the first storage device can write the data obtained from the processing operation to a segment of physical storage space on a storage device indicated by that second physical address. When there are multiple second physical addresses, the first storage device can write the data obtained from the processing operation to physical storage spaces in different storage devices, or to different physical storage spaces within the same storage device. For example, if the data migration task is a partial migration task (such as a write-back cache task), the index subsystem 202 can generate a partial migration task to instruct the target data to be written to multiple storage devices in the storage system 10. In this case, the migration request generated based on the partial migration task may include second physical addresses on multiple storage devices used to back up the target data. Thus, when the first storage device executes the migration request, it can transfer the target data to the corresponding multiple storage devices for storage according to the multiple second physical addresses.

[0096] Furthermore, the number of first physical addresses included in the migration request can be one or more.

[0097] 1. The number of first physical addresses is one.

[0098] At this point, the data migration task can be either a partial relocation task or a complete relocation task. The first storage device can perform a data backup operation on the data indicated by the first physical address and save the obtained data to the storage location indicated by the second physical address.

[0099] 2. There are multiple first physical addresses.

[0100] At this point, the data migration task can be, for example, a collaborative migration task or a partial migration task.

[0101] For example, in an EC reconstruction scenario (where the data migration task is a collaborative migration task), there can be multiple first physical addresses. In this case, different first physical addresses can indicate physical storage space on different storage devices.

[0102] For example, suppose data is stored in storage system 10 using EC technology across K+1 storage devices, where K is an integer greater than 1. When data is lost on one of the storage devices, the persistence subsystem 203 can generate a collaborative migration task to instruct the recovery of the lost data using data from the remaining K storage devices. The migration request generated based on this collaborative migration task can include the first physical addresses of the remaining data when it was stored on each of the K storage devices. Thus, when the first storage device executes the migration request, it can retrieve the target data from each of the corresponding K storage devices based on the K first physical addresses carried in the migration request, reconstruct the data to be recovered (i.e., reconstruct the lost data) based on the target data retrieved from the K storage devices, and further save the reconstructed data on the first storage device.

[0103] For example, in a scenario where data is migrated based on EC technology (the data migration task can be a partial migration task), there can be multiple first physical addresses. In this case, different first physical addresses can indicate multiple different physical storage spaces on the same storage device. The data in each physical storage space is the data located in the storage device from the multiple stripes included in the first logical storage unit.

[0104] Furthermore, when the processing operation in the migration request is specifically a migration operation, since the migration request is transformed from a data migration task based on logical storage unit level semantics, the first storage device, when executing the migration request, still migrates the target data indicated by the first physical address according to the data processing logic at the logical storage unit level. When there are multiple first physical addresses, the first storage device typically migrates the data in the physical storage space indicated by each first physical address one by one. This results in low migration efficiency for data in different physical storage spaces, thus leading to low execution efficiency of the data migration task. Moreover, when storage is based on EC technology, if data is migrated according to the data processing logic at the logical storage unit level, the first storage device typically first reads the data from each storage device participating in EC storage, performs EC calculations on the data, and then splits the data and verification data to different storage devices for storage. This generates a large converged read traffic (i.e., data from other storage devices converges to the first storage device).

[0105] Therefore, as shown in Figure 1, the task processing device 200 may further include a data-aware task orchestration (DATO) module. This DATO module can generate a data view corresponding to the first storage device based on the storage location of the target data on the first storage device. This data view describes the location distribution of the target data on the first storage device. Based on the data view corresponding to the first storage device, a migration request that improves task execution efficiency is generated. Thus, the DATO module can provide the migration request to the first storage device so that the first storage device can respond to the migration request and process the target data. In this way, the DATO module plans and controls the migration process of data indicated by different first physical addresses based on the data view, thereby optimizing the efficiency of data migration between different physical storage spaces, i.e., improving the execution efficiency of the data migration task. For example, assuming the first logical storage unit is a chunk, the logical address information in the data migration task and the physical address information in the migration request are shown in Figure 6, which can be used to describe the storage location of the target data on the storage device. The DATO module can then generate the data view shown in Figure 6 based on the logical address information in the data migration task and the converted physical address information. Furthermore, when migrating data stored based on EC technology, the first storage device can migrate data belonging to the first logical storage unit on the first storage device according to the data view, which can also avoid generating large aggregate read traffic.

[0106] The following examples illustrate this concept using specific data migration scenarios.

[0107] Scenario 1: Data is backed up and stored in storage system 10, meaning that a copy of the same data can be stored on multiple storage devices. When it is necessary to migrate target data belonging to the first logical storage unit on the first storage device to multiple other storage devices, the task processing device 200 (the DATO module in it) can determine the data to be migrated to multiple first physical addresses in the same storage device according to the data view corresponding to the first storage device (describing the location distribution of the target data on the first storage device), generate migration requests for the data indicated by the multiple first physical addresses, and send the migration requests to the first storage device so as to instruct the first storage device to perform batch backup of the data indicated by the multiple first physical addresses.

[0108] For example, as shown in Figure 7, assuming the first storage device is storage device 10, and the target data belonging to the first logical storage unit includes data in multiple first physical addresses corresponding to chunks 8, 9, and 10 (the physical storage space is not contiguous), where the target data in chunks 8 and 10 needs to be backed up in storage devices 1, 3, and 5, and the target data in chunk 9 needs to be backed up in storage devices 2, 4, and 6 (e.g., different data are backed up in different storage devices based on a load balancing strategy). After obtaining the migration request, the task processing device 200 can generate a data view as shown in Figure 7 based on the location distribution of the target data on storage device 10, and generate migration request 1 (or control command) and migration request 2 as shown in Figure 7 based on this data view. Migration request 1 instructs storage device 10 to migrate 4KB of data in chunk 9 to storage devices 2, 4, and 6. Migration request 2 instructs storage device 10 to batch back up two 4KB target data in chunk 8 and 8KB target data in chunk 10 to storage devices 1, 3, and 5, as shown in Figure 7. Thus, for the target data indicated by these three physical addresses, the task processing device 200 can control the storage device 10 to perform batch backups to reduce the number of I / O operations generated during the data migration process and improve data migration efficiency.

[0109] Scenario 2: Data is stored in storage system 10 using EC (Elastic Compute Service) technology. In this scenario, the target data in the first logical storage unit can include multiple stripes of data, and each stripe's data is distributed across multiple storage devices. That is, each storage device can store a portion of the data for that stripe. Therefore, the data view corresponding to each storage device is a data view generated based on the location distribution of the target data on that storage device. Assume that in storage system 10, data is stored using (m+n) storage devices based on EC technology, where m is an integer greater than 1 and n is a positive integer. When target data needs to be migrated to other storage devices for storage, for each storage device involved in storing the target data (hereinafter referred to as the first storage device), the task processing unit 200 (the DATO module in it) can determine m data points (or user data / valid data) from the target data based on the data view corresponding to the first storage device (describing the location distribution of data on the first storage device among multiple stripes corresponding to the target data). These m data points are the data located on the first storage device among the multiple stripes included in the target data. A migration request is then generated for these m data points and sent to the first storage device. In this way, the first storage device can, based on the received migration request, specifically based on the processing operation carried in the migration request, perform EC-based calculations to generate n checksums for the m data points. In practical applications, the first storage device can be configured with hardware for EC calculations, and the first storage device can utilize this hardware to perform operations such as generating EC calculations for the m data points to obtain n checksums. Alternatively, the first storage device can be configured with software to implement EC (Extended EC) technology. This software allows the first storage device to perform EC calculations on m data points to obtain n checksums. Then, the first storage device splits the different data (including user data and checksums) within the stripe and stores them on different storage devices. This avoids first aggregating stripe data from multiple storage devices (the storage devices where the data resides before migration) and then distributing the stripe data across different storage devices (the storage devices where the data resides after migration). This avoids aggregating read traffic, improving data migration efficiency and reducing resource overhead. For the remaining data in the target data, excluding the m data points, the first storage device can also perform data migration in a similar manner based on the migration request. Furthermore, when the remaining data is less than m, it can be padded with zeros to construct m data points before performing data migration in a similar manner.

[0110] For example, as shown in Figure 8, suppose that in storage system 10, target data is stored using storage devices 0, 1, 2, and 3, and checksum data corresponding to the target data is stored using storage devices 4 and 5 (i.e., using a 4+2 EC storage configuration). The data view corresponding to the target data stored on storage device 0 is shown in Figure 8. Then, for storage device 0, the task processing unit 200 can determine the data in four storage regions from the target data on storage device 0 based on this data view, as shown in Figure 8, and generate migration requests for the data in these four storage regions. For example, it can generate the migration request shown in Figure 8 and send the migration request to storage device 0. Storage device 0 can then generate checksum data of two storage region sizes (each storage region has the same specifications, such as 4KB) for the data in the four storage regions based on EC technology, according to the received migration request. Thus, the task processing device 200 can generate a new stripe based on the data in the four storage areas and the check data of the size of two storage areas, and split the data of different parts of the new stripe to be stored in six other storage devices (such as storage devices 6 to 11), as shown in Figure 8. Each storage device can store the same amount of data. For each storage device storing the target data, the task processing device 200 can generate a corresponding migration request for the storage device in the manner described above, and instruct the storage device to distribute the target data stored on it to different storage devices by sending the migration request to the storage device. In this process, since the storage device generates check data for the target data locally and saves the check data on other storage devices, the newly generated check data can be used to ensure the accuracy and reliability of the migrated target data in the storage system 10 after the target data is migrated. At the same time, during the migration of target data, there is no need to perform the process of reading target data from storage devices 0 to 3, thereby avoiding the accumulation of read traffic. In practical applications, the read traffic saved based on the above process can reach 50%. It should be noted that in this embodiment, storage devices 4 and 5 are used to store verification data separately. In actual applications, a single storage device can store not only valid data / user data from a portion of the stripes, but also verification data from other stripes. In this case, the storage device can also refer to the above method, according to the data view, combine the data from multiple stripes on the storage device into a single stripe based on EC technology, and then split the stripe into multiple different storage devices for storage.

[0111] Furthermore, for storage devices 4 and 5 that store the verification data corresponding to the target data, the task processing device 200 can, in accordance with the above method, instruct storage devices 4 and 5 to also transfer the initially generated verification data corresponding to the target data to other storage devices for storage. Alternatively, if the accuracy and reliability of the target data have already been ensured using the newly generated verification data, the task processing device 200 may not instruct storage devices 4 and 5 to save the initially generated verification data stored thereon to other storage devices. For example, the task processing device 200 may instruct storage devices 4 and 5 to delete the verification data stored thereon and release the storage space occupied by the verification data.

[0112] In this embodiment, the task processing device 200 performs processing operations on target data in the first logical storage unit as an example. In actual application scenarios, when there are east-west traffic tasks in the storage system 10 that generate data in other logical storage units, the task processing device 200 can refer to the above method to convert the task with logical storage unit level semantics into a migration request with storage device level semantics including physical address, and send the migration request to the second storage device for execution, thereby improving the efficiency of task execution in the storage system 10.

[0113] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0114] The task processing method provided by the embodiments of this application has been described above with reference to Figures 1 to 8. Next, the structure of the task processing device and computing device provided by the embodiments of this application will be described with reference to the accompanying drawings.

[0115] Referring to Figure 9, a schematic diagram of a task processing device is shown. The task processing device 900 shown in Figure 9 is applied to a storage system, which includes the task processing device 900 and multiple storage devices for persistently storing data.

[0116] As shown in Figure 9, the task processing device 900 includes:

[0117] The acquisition module 901 is used to receive a data migration task. The data migration task includes a source logical address and a destination logical address. The data migration task is used to migrate the target data indicated by the source logical address to the destination logical address. The source logical address belongs to the first logical storage unit and the destination logical address belongs to the second logical storage unit. The first logical storage unit corresponds to the first physical storage space in multiple storage devices and the second logical storage unit corresponds to the second physical storage space in multiple storage devices.

[0118] The conversion module 902 is used to convert the source logical address into the first physical address in the first physical storage space and to convert the destination logical address into the second physical address in the second physical storage space.

[0119] The generation module 903 is used to generate a migration request, which carries a first physical address and a second physical address.

[0120] The communication module 904 is used to send a migration request to a first storage device among a plurality of storage devices. The migration request is used to instruct the first storage device to migrate target data from a first physical address to a second physical address.

[0121] In one possible implementation, a data migration task is used to instruct target data to be migrated from a source logical address to a destination logical address;

[0122] Alternatively, a data migration task is used to instruct the target data to be backed up to the destination logical address;

[0123] Alternatively, a data migration task can be used to instruct the reconstruction of first data based on EC (erasure coding) technology according to the target data, and to store the first data in a second physical address.

[0124] In one possible implementation, the generation module 903 is further configured to:

[0125] Determine the business type corresponding to the data migration task;

[0126] When the business type is the first type, the migration request is used to instruct the first storage device to migrate the target data from the first physical address to the second physical address;

[0127] When the business type is type 2, the migration request is used to instruct the first storage device to back up the target data to the second physical address;

[0128] When the business type is the third type, the migration request is used to instruct the first storage device to reconstruct the data based on EC technology according to the target data of the first physical address.

[0129] In one possible implementation, the target data is located in the first storage device, and the generation module 903 is further configured to generate a data view corresponding to the first storage device based on the storage location of the target data on the first storage device. The data view is used to describe the location distribution of the target data on the first storage device.

[0130] Therefore, when generating a migration request, the generation module 903 is specifically used to generate a migration request based on the data view corresponding to the first storage device.

[0131] In one possible implementation, a data migration task is used to instruct the migration of target data from a source logical address to a destination logical address, and a migration request is used to instruct the migration of target data from a storage location indicated by a first physical address to a storage location indicated by a second physical address.

[0132] In one possible implementation, the target data is stored in the storage system based on EC technology;

[0133] The migration request is specifically used to indicate:

[0134] Determine m data points from the target data, where m is an integer greater than 1;

[0135] Based on EC technology, n verification data are generated for m data, where n is a positive integer;

[0136] For a stripe consisting of m data points and n check data points, the different data points in the stripe are split and stored on different storage devices.

[0137] In one possible implementation, a data migration task is used to instruct the target data to be backed up to a destination logical address;

[0138] Therefore, when generating a migration request, the generation module 903 is specifically used for:

[0139] Based on the data view corresponding to the first storage device, identify multiple second data items in the target data that are migrated to the same physical address in the second physical storage space;

[0140] For multiple secondary data sets, a migration request is generated, which instructs that multiple secondary data sets be backed up in batches.

[0141] In one possible implementation, the target data is stored in the storage system based on EC technology, and the migration request is used to instruct the first storage device to reconstruct the first data according to the target data and store the first data at a second physical address in the first storage device.

[0142] In one possible implementation, multiple storage devices exchange data based on an interconnect bus;

[0143] Alternatively, multiple storage devices can interact with each other based on network interface cards (NICs), smart NICs, or data processing units (DPUs).

[0144] Since the task processing device 900 shown in FIG9 corresponds to the task processing device 200 in the embodiment shown in FIG3 above, the specific implementation method and technical effects of the task processing device 900 shown in FIG9 can be found in the relevant descriptions in the embodiment shown in FIG3 above, and will not be repeated here.

[0145] Figure 10 is a schematic diagram of the hardware structure of a computing device 1000 provided in this application. The computing device 1000 can, for example, implement the task processing device 200 in the embodiment shown in Figure 3 above.

[0146] As shown in Figure 10, the computing device 1000 includes a processor 1001, a memory 1002, and a communication interface 1003. The processor 1001, memory 1002, and communication interface 1003 communicate via a bus 1004, or via wireless transmission or other means. The memory 1002 stores instructions, and the processor 1001 executes the instructions stored in the memory 1002. Further, the computing device 1000 may also include a memory unit 1005, which is connected to the processor 1001, the storage medium 1002, and the communication interface 1003 via the bus 1004. The memory 1002 stores program code, and the processor 1001 can perform the following operations using the program code stored in the memory 1002:

[0147] A data migration task is received. The data migration task includes a source logical address and a destination logical address. The data migration task is used to migrate the target data indicated by the source logical address to the destination logical address. The source logical address belongs to a first logical storage unit, and the destination logical address belongs to a second logical storage unit. The first logical storage unit corresponds to a first physical storage space in a plurality of storage devices, and the second logical storage unit corresponds to a second physical storage space in the plurality of storage devices. The plurality of storage devices are used for persistent storage of data.

[0148] The source logical address is converted into a first physical address in the first physical storage space, and the destination logical address is converted into a second physical address in the second physical storage space;

[0149] Generate a migration request, the migration request carrying the first physical address and the second physical address;

[0150] The migration request is sent to a first storage device among the plurality of storage devices, the migration request being used to instruct the first storage device to migrate the target data from the first physical address to the second physical address.

[0151] It should be understood that in this embodiment, the processor 1001 can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0152] The memory 1002 may include read-only memory and random access memory, and provides instructions and data to the processor 1001. The memory 1002 may also include non-volatile random access memory.

[0153] The memory 1002 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0154] The communication interface 1003 is used to communicate with other devices connected to the computing device 1000. The bus 1004 may include a data bus, as well as a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 1004 in the figure.

[0155] It should be understood that the computing device 1000 according to the embodiments of this application may correspond to the task processing device 200 in the embodiments of this application, and may correspond to the method executed by the task processing device 200 in the method shown in FIG3 of the embodiments of this application. The above and other operations and / or functions implemented by the computing device 1000 are respectively to implement the flow of the corresponding method in FIG3. For the sake of brevity, they will not be described in detail here.

[0156] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the above-described task processing method.

[0157] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0158] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0159] The computer program product can be a software installation package. When any of the aforementioned task processing methods is required, the computer program product can be downloaded and executed on a computing device.

[0160] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0161] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0162] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0163] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A task processing method, characterized in that, The method is applied to a storage system, the storage system including multiple storage devices, the multiple storage devices being used for persistent storage of data, the method comprising: A data migration task is received. The data migration task includes a source logical address and a destination logical address. The data migration task is used to migrate the target data indicated by the source logical address to the destination logical address. The source logical address belongs to a first logical storage unit, and the destination logical address belongs to a second logical storage unit. The first logical storage unit corresponds to a first physical storage space in the plurality of storage devices, and the second logical storage unit corresponds to a second physical storage space in the plurality of storage devices. The source logical address is converted into a first physical address in the first physical storage space, and the destination logical address is converted into a second physical address in the second physical storage space; Generate a migration request, the migration request carrying the first physical address and the second physical address; The migration request is sent to a first storage device among the plurality of storage devices, the migration request being used to instruct the first storage device to migrate the target data from the first physical address to the second physical address.

2. The method according to claim 1, characterized in that, The data migration task is used to instruct the target data to be migrated from the source logical address to the destination logical address; Alternatively, the data migration task can be used to instruct the target data to be backed up to the destination logical address; Alternatively, the data migration task may be used to instruct data reconstruction based on erasure coding (EC) technology according to the target data.

3. The method according to claim 2, characterized in that, The method further includes: Determine the business type corresponding to the data migration task; When the service type is the first type, the migration request is used to instruct the first storage device to migrate the target data of the first physical address to the second physical address; When the service type is the second type, the migration request is used to instruct the first storage device to back up the target data to the second physical address; When the service type is the third type, the migration request is used to instruct the first storage device to reconstruct the first data based on the target data of the first physical address using EC technology, and store the first data to the second physical address.

4. The method according to claim 2 or 3, characterized in that, The target data is located in the first storage device, and the method further includes: Based on the storage location of the target data on the first storage device, a data view corresponding to the first storage device is generated, and the data view is used to describe the location distribution of the target data on the first storage device; Then, generating the migration request includes: The migration request is generated based on the data view corresponding to the first storage device.

5. The method according to claim 4, characterized in that, The target data is stored in the storage system based on the EC technology; Therefore, the migration request is specifically used to indicate: Determine m data points from the target data, where m is an integer greater than 1; Based on the EC technology, n verification data are generated for the m data, where n is a positive integer; For a stripe consisting of m data points and n verification data points, the different data points in the stripe are split and stored on different storage devices.

6. The method according to claim 4, characterized in that, The data migration task is used to instruct the target data to be backed up to the destination logical address; The step of generating the migration request based on the data view corresponding to the first storage device includes: Based on the data view corresponding to the first storage device, determine multiple second data items in the target data that have been migrated to the same physical address in the second physical storage space; For the plurality of second data, a migration request is generated, which is used to instruct the plurality of second data to be backed up in batch.

7. The method according to any one of claims 1 to 6, characterized in that, The multiple storage devices interact with each other via an interconnect bus. Alternatively, the multiple storage devices may interact with each other based on a network interface card (NIC), a smart NIC, or a data processing unit (DPU).

8. A task processing device, characterized in that, The task processing device is applied to a storage system, which includes multiple storage devices for persistently storing data. The task processing device includes: The acquisition module is used to receive a data migration task, the data migration task includes a source logical address and a destination logical address, the data migration task is used to migrate the target data indicated by the source logical address to the destination logical address, the source logical address belongs to a first logical storage unit, the destination logical address belongs to a second logical storage unit, the first logical storage unit corresponds to a first physical storage space in the plurality of storage devices, and the second logical storage unit corresponds to a second physical storage space in the plurality of storage devices; The conversion module is used to convert the source logical address into a first physical address in the first physical storage space, and to convert the destination logical address into a second physical address in the second physical storage space; The generation module is used to generate a migration request, wherein the migration request carries the first physical address and the second physical address; A communication module is configured to send the migration request to a first storage device among the plurality of storage devices, the migration request being used to instruct the first storage device to migrate the target data from the first physical address to the second physical address.

9. The apparatus according to claim 8, characterized in that, The data migration task is used to instruct the target data to be migrated from the source logical address to the destination logical address; Alternatively, the data migration task can be used to instruct the target data to be backed up to the destination logical address; Alternatively, the data migration task may be used to instruct data reconstruction based on erasure coding (EC) technology according to the target data.

10. The apparatus according to claim 9, characterized in that, The generation module is further configured to: Determine the business type corresponding to the data migration task; When the service type is the first type, the migration request is used to instruct the first storage device to migrate the target data of the first physical address to the second physical address; When the service type is the second type, the migration request is used to instruct the first storage device to back up the target data to the second physical address; When the service type is the third type, the migration request is used to instruct the first storage device to reconstruct the first data based on the target data of the first physical address using EC technology, and store the first data to the second physical address.

11. The apparatus according to claim 9 or 10, characterized in that, The target data is located in the first storage device. The generation module is further configured to generate a data view corresponding to the first storage device based on the storage location of the target data on the first storage device. The data view is used to describe the location distribution of the target data on the first storage device. Therefore, when generating a migration request, the generation module is specifically used to generate the migration request based on the data view corresponding to the first storage device.

12. The apparatus according to claim 11, characterized in that, The target data is stored in the storage system based on the EC technology; The migration request is specifically used to indicate: Determine m data points from the target data, where m is an integer greater than 1; Based on the EC technology, n verification data are generated for the m data, where n is a positive integer; For a stripe consisting of m data points and n verification data points, the different data points in the stripe are split and stored on different storage devices.

13. The apparatus according to claim 11, characterized in that, The data migration task is used to instruct the target data to be backed up to the destination logical address; Therefore, when generating the migration request, the generation module is specifically used for: Based on the data view corresponding to the first storage device, determine multiple second data items in the target data that have been migrated to the same physical address in the second physical storage space; For the plurality of second data, a migration request is generated, which is used to instruct the plurality of second data to be backed up in batch.

14. The apparatus according to any one of claims 8 to 13, characterized in that, The multiple storage devices interact with each other via an interconnect bus. Alternatively, the multiple storage devices may interact with each other based on a network interface card (NIC), a smart NIC, or a data processing unit (DPU).

15. A storage system, characterized in that, The storage system includes a task processing device and multiple storage devices, the multiple storage devices being used for persistent storage of data, and the multiple storage devices including a first storage device; The task processing device is configured to perform the method as described in any one of claims 1 to 7; The first storage device is further configured to execute a migration request to migrate target data from a first physical address to a second physical address.

16. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the steps of the method as described in any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computing device, cause the computing device to perform the steps of the method as described in any one of claims 1 to 7.

18. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 1 to 7.

Images