Data processing method, apparatus and device, medium, product

Abstract

Embodiments of the present application disclose a data processing method, device and equipment, medium and product. The method is applied to a distributed system. Each computing power cluster device in the distributed system comprises at least one computing power module. The method comprises the following steps: receiving a write request sent by a first computing power module, wherein the write request carries a backup source address and data volume information of first data; dividing each computing power module in the distributed system into multiple failure domain levels based on a topological distance between the first computing power module and each computing power module in the distributed system; determining a backup allocation strategy for each failure domain level according to the data volume information of the first data and storage state information of each computing power module in each failure domain level; and backing up the first data in each failure domain level according to the backup allocation strategy for each failure domain level based on the backup source address of the first data. The embodiments of the present application can achieve efficient and secure data protection in a complex distributed environment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of data backup technology, specifically to a data processing method, a data processing device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In distributed systems, data backup is a disaster recovery technology used to address unforeseen risks such as hardware failures, network anomalies, and equipment downtime, ensuring data reliability. Traditional data backup solutions typically employ fixed replica placement rules. For example, the number of replicas is pre-configured statically, and replicas are fixedly mapped and stored in specific physical devices, racks, or data centers according to pre-configured settings. Storage routing and synchronization rules also remain static. However, with the deep integration of cloud computing, big data, and artificial intelligence technologies, modern distributed systems exhibit significant multi-layered topologies (e.g., computing cluster devices - computing modules - storage units) and dynamic fluctuations in storage resources (e.g., real-time changes in storage load, which can easily lead to overload of some devices, write failures, or uneven replica distribution). The rigid replica placement rules of traditional data backup solutions are no longer suitable for modern distributed systems. Therefore, there is an urgent need to design a new data backup solution for distributed systems to achieve efficient and secure data protection in complex distributed environments. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, medium, and product, and designs a new data backup scheme for distributed systems, which can achieve efficient and secure data protection in complex distributed environments.

[0004] On one hand, embodiments of this application provide a data processing method applied to a distributed system, which includes multiple computing power cluster devices, each of which includes at least one computing power module. The method includes: Receive a write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data. Based on the topological distance between the first computing power module and each computing power module in the distributed system, each computing power module in the distributed system is divided into multiple fault domain levels. Based on the data volume information of the first data and the storage status information of each computing module in each fault domain level, a backup allocation strategy for each fault domain level is determined. Based on the backup source address of the first data, the first data is backed up in each fault domain level according to the backup allocation strategy of each fault domain level.

[0005] On one hand, embodiments of this application provide a data processing apparatus applied to a distributed system, the distributed system including multiple computing power cluster devices, each computing power cluster device including at least one computing power module, the apparatus including: The receiving unit is used to receive the write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data. The processing unit is used to divide each computing power module in the distributed system into multiple fault domain levels based on the topological distance between the first computing power module and each computing power module in the distributed system. The processing unit is also used to determine the backup allocation strategy for each fault domain level based on the data volume information of the first data and the storage status information of each computing module in each fault domain level. The processing unit is also used to back up the first data in each fault domain level according to the backup source address of the first data and the backup allocation strategy of each fault domain level.

[0006] In one embodiment, the processing unit divides the computing modules in the distributed system into multiple fault domain levels based on the topological distance between the first computing module and each computing module in the distributed system, including: Obtain the topology information of the distributed system; The topological distance between each computing module in a multi-computing cluster and the first computing module is determined based on topological structure information; one fault domain level corresponds to one topological distance. Based on the correspondence between topological distance and fault domain level, each computing module in multiple computing power cluster devices is divided into multiple fault domain levels.

[0007] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processing unit determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the i-th computing module and the first computing module are the same computing module, then the topological distance between the i-th computing module and the first computing module is determined as the first distance; The processing unit divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the first distance and the first fault domain level, the i-th computing power module unit is assigned to the first fault domain level.

[0008] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processing unit determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the i-th computing power module and the first computing power module are not the same computing power module, and the i-th computing power module and the first computing power module both belong to the first computing power cluster device, then the topology distance between the i-th computing power module and the first computing power module is determined as the second distance; The processing unit divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the second distance and the second fault domain level, the i-th computing power module unit is assigned to the second fault domain level.

[0009] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processing unit determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the second computing power cluster device, and the first computing power cluster device and the second computing power cluster device are two computing power cluster devices located in the same region, then the topology distance between the i-th computing power module and the first computing power module is determined as the third distance. The processing unit divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the third distance and the third fault domain level, the i-th computing power module unit is assigned to the third fault domain level.

[0010] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processing unit determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the third computing power cluster device, and the first computing power cluster device and the third computing power cluster device are two computing power cluster devices located in different regions, then the topology distance between the i-th computing power module and the first computing power module is determined as the fourth distance. The processing unit divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the fourth distance and the fourth fault domain level, the i-th computing power module unit is assigned to the fourth fault domain level.

[0011] In one embodiment, each computing power module includes at least one storage unit, and each storage unit is divided into at least one data block according to a preset partitioning granularity N. The storage status information of the computing power module includes the remaining capacity information and allocation status information of each data block in the computing power module. Each data block contains N data units. The allocation status information of the data block is used to indicate whether each data unit in the data block has been allocated. Any one of the multiple fault domain levels is represented as fault domain level j, where j and N are positive integers. Based on the data volume information of the first data and the storage status information of each computing module in fault domain level j, the processing unit determines the backup allocation strategy for fault domain level j, including: From the various computing power modules at fault domain level j, determine the computing power module j used to back up the first data; Based on the remaining capacity information of each data block in computing module j and the data volume information of the first data, the data block j used to back up the first data is determined from each data block of computing module j; Based on the allocation status information of data block j and the data volume information of the first data, the backup allocation strategy of fault domain level j is determined. The backup allocation strategy of fault domain level j is used to indicate the data unit sequence j in data block j used to back up the first data. Wherein, when fault domain level j is the first fault domain level, computing module j is the first computing module; when fault domain level j is the second fault domain level, computing module j is at least one computing module belonging to the same first computing cluster device as the first computing module; when fault domain level j is the third fault domain level, computing module j is each computing module in the fourth computing cluster device, which is determined from each computing cluster device located in the same region as the first computing cluster device to which the first computing module belongs, according to a first decision strategy, the first decision strategy including at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy; when fault domain level j is the fourth fault domain level, computing module j is each computing module in the fifth computing cluster device, which is determined from each computing cluster device located in a different region from the first computing cluster device to which the first computing module belongs, according to a second decision strategy, the second decision strategy including at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy, network latency strategy, topology awareness strategy.

[0012] In one embodiment, a storage unit corresponds to a state tree, and each state tree contains at least one leaf node. A leaf node in each state tree corresponds to a data block in the corresponding storage unit. Each leaf node is used to record the remaining capacity information of the corresponding data block. Each parent node in each state tree is used to record the maximum remaining capacity information among all the remaining capacity information recorded by all child nodes under the corresponding parent node. A preset number of child nodes in each state tree are linked to the same parent node. Based on the remaining capacity information of each data block in computing module j and the data volume information of the first data, the processing unit determines the data block j used to back up the first data from the data blocks of computing module j, including: Based on the data volume information of the first data, traverse the state tree corresponding to each storage unit in the computing power module j; During the traversal, if the remaining capacity information recorded by the traversed node does not match the data volume information of the first data, then skip all descendant nodes under the traversed node until the target leaf node whose recorded remaining capacity information matches the data volume information of the first data is reached. The data block corresponding to the target leaf node is determined as data block j used to back up the first data.

[0013] In one embodiment, multiple fault domain levels have priorities, each fault domain level corresponds to a topological distance, and the smaller the corresponding topological distance, the higher the priority of the fault domain level; the processing unit is also used for: If the target leaf node is not traversed, the next fault domain level with a lower priority than fault domain level j is determined, and the data block j used to back up the first data is determined from each data block of the next fault domain level.

[0014] In one embodiment, each leaf node is also used to record the allocation status information of the corresponding data block, and each leaf node is bound to the logical address of the corresponding data block. Based on the allocation status information of data block j and the data volume information of the first data, the processing unit determines the backup allocation strategy for fault domain level j, including: Based on the data volume information of the first data and the allocation status information of data block j, determine the data unit sequence j in data block j used to back up the first data; the data unit sequence j contains one or more consecutive and idle data units; The intra-block offset information of data unit sequence j in data block j and the logical address bound to the target leaf node are determined as the backup destination address of fault domain level j. The backup allocation strategy for fault domain level j is generated based on the backup destination address of fault domain level j.

[0015] In one embodiment, the processing unit is further configured to: A space allocation instruction is issued to each computing power cluster device. The space allocation instruction is used to instruct each computing power cluster device to divide each storage unit in each computing power module into at least one data block according to the preset partitioning granularity N, and to generate the logical address of each data block based on the preset mapping function and the physical address of each data block. Receive the logical address of each data block in each storage unit returned by each computing cluster device; Based on the logical address of at least one data block in each storage unit and the preset partitioning granularity N, at least one leaf node of each storage unit is initialized. The logical address of a data block is bound to an initialized leaf node. The remaining capacity information and allocation status information recorded in an initialized leaf node are determined based on the preset partitioning granularity N. Starting from at least one leaf node of each storage unit, a preset number of child nodes are linked to the same parent node, and the maximum remaining capacity information recorded in the remaining capacity information of each child node linked to the same parent node is added to the corresponding parent node to obtain the state tree corresponding to each storage unit.

[0016] In one embodiment, the processing unit is further configured to: When the first data is successfully backed up in data block j, the state tree corresponding to the storage unit to which data block j belongs is updated.

[0017] In one embodiment, each computing power module further includes at least one computing unit, the write request is sent by the first computing unit in the first computing power module, the backup source address of the first data points to the first computing unit, the backup allocation strategy of fault domain level j includes the backup destination address of fault domain level j, and the backup destination address of fault domain level j points to data unit sequence j and data block j. Based on the backup source address of the first data, the processing unit backs up the first data in fault domain level j according to the backup allocation strategy of fault domain level j, including: A backup instruction is issued to the first computing unit. The backup instruction is used to instruct the first computing unit to establish a first transmission path from the first computing unit to the storage unit to which the data block j belongs, based on the backup destination address of the fault domain level j and the backup source address of the first data, and to write the first data into the data unit sequence j in the data block j through the first transmission path. When the first computing unit and the storage unit to which data block j belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the first computing unit and the storage unit to which data block j belong belong to different computing modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

[0018] In one embodiment, the backup allocation policy for each fault domain level includes the backup destination address for that fault domain level. The backup destination address in each fault domain level's backup allocation policy is associated with the identification information of the first data. The receiving unit is further configured to: Receive a read request sent by the second computing power module. The read request carries the identification information of the first data and the address of the requesting end. The processing unit is also used for: Query multiple backup destination addresses associated with the identification information of the first data, and determine the target backup destination address from the multiple backup destination addresses; Based on the target backup destination address and the requesting end address, the first data is read into the second computing power module.

[0019] In one embodiment, the processing unit determines the target backup destination address from a plurality of backup destination addresses, including: Obtain the computing power module corresponding to each backup destination address; Determine the topological distance between the computing power modules corresponding to multiple backup destination addresses and the second computing power module; The backup destination address with the smallest topological distance among multiple backup destination addresses is determined as the target backup destination address.

[0020] In one embodiment, the read request is sent by the second computing unit in the second computing power module, the requesting address points to the second computing unit, and the target backup destination address points to the target data unit sequence and the target data block; The processing unit reads the first data into the second computing module based on the target backup destination address and the requesting end address, including: A read instruction is sent to the second computing unit. The read instruction is used to instruct the second computing unit to establish a second transmission path from the storage unit to which the target data block belongs to the second computing unit based on the target backup destination address and the requesting end address, and to read the first data from the target data unit sequence in the target data block to the second computing unit through the second transmission path. When the second computing unit and the storage unit to which the target data block belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the second computing unit and the storage unit to which the target data block belongs belong to different computing modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

[0021] On one hand, embodiments of this application provide a computer device, which includes: Processor, memory, and network interface; The processor is connected to the memory and the network interface. The network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to implement the above data processing method.

[0022] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including computer instructions adapted to be loaded by a processor and executed by the above-described data processing method.

[0023] On the one hand, embodiments of this application provide a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, the above-described data processing method is implemented.

[0024] This application embodiment can receive a write request sent by a first computing power module. The write request is used to request the backup of first data in the distributed system. The write request carries the backup source address and data volume information of the first data. The backup source address and data volume information can provide a deterministic input basis for subsequent topology awareness and storage resource scheduling, avoiding blind allocation of storage resources. Based on the topological distance between the first computing power module (the computing power module pointed to by the backup source address of the first data) and each computing power module in the distributed system, each computing power module in the distributed system is divided into multiple fault domain levels. By constructing a multi-level fault domain isolation mechanism based on topological distance, data replicas can be distributed and stored in computing power modules at different topological distances, improving the system's ability to resist regional faults (such as computing power module failure, rack power failure, and switch failure). Based on the data volume information of the first data and the storage status information of each computing power module in each fault domain level, a backup allocation strategy for each fault domain level is determined. By combining storage status information and data volume for dynamic decision-making, the storage load imbalance problem caused by traditional static strategies can be solved, optimizing storage space utilization and read / write response speed. Based on the backup source address of the first data, the first data is backed up in each fault domain level according to the backup allocation strategy of each fault domain level. Since the replicas on different fault domain levels are stored in computing power modules that are topologically isolated from each other, the replicas of other fault domain levels are not affected when a local failure occurs in any fault domain level, thereby significantly enhancing data fault tolerance and system high availability. It can be seen that the embodiments of this application design a new data backup scheme for distributed systems, which can achieve data protection that balances storage resource utilization and fault isolation capabilities in complex distributed environments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a data backup scheme provided in an embodiment of this application; Figure 2 This is a schematic diagram of the architecture of a distributed system provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 ; Figure 4 This is a schematic diagram of a state tree construction process provided in an embodiment of this application; Figure 5 This is a flowchart illustrating a copy allocation decision method provided in an embodiment of this application; Figure 6 This is a schematic diagram of a data transmission process provided in an embodiment of this application; Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 ; Figure 8 This is a schematic diagram of a data reading process provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0026] To facilitate a better understanding of the technical solutions proposed in this application, the technical terms involved in this application will be explained below.

[0027] (1) Distributed system: refers to a computing system composed of multiple independent computing devices (such as servers) connected through a network, and providing computing, storage or input / output services to the outside world through a unified coordination mechanism. Distributed systems can allocate computing and storage tasks among multiple computing devices, aiming to solve the bottlenecks of single computing devices in terms of processing power, storage space or reliability. With the deep application of cloud computing, big data and artificial intelligence technologies, the computing devices in distributed systems are usually computing power cluster devices.

[0028] (2) Computing cluster equipment: refers to a collection of physical hardware with high-density computing power and local storage capabilities, such as supernode equipment. Supernode equipment is a high-performance computing device designed for large-scale artificial intelligence (AI) training, inference, or high-performance computing scenarios. Computing cluster equipment can accommodate multiple computing modules.

[0029] (3) Computing Module (Tray): Refers to the physical functional partition within a computing cluster device, which can be independently powered on, disconnected, and replaced, such as a pluggable computing tray in a server. A computing module typically contains at least one computing unit and at least one storage unit. A computing unit refers to the processor chip responsible for performing numerical calculations, matrix operations, inference training tasks, etc., such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Processing Unit). For example, in a GPU supernode device, each Tray contains several GPUs as computing units; in an NPU supernode device, each Tray contains several NPUs as computing units. A storage unit refers to the non-volatile storage medium mounted next to the computing unit for model loading, temporary data caching, etc. For example, each Tray contains several Non-Volatile Memory Express (NVMe) solid-state drives as storage units. Different Trays within the same computing cluster device are usually interconnected via high-speed buses such as NVLink and NVMe-OF.

[0030] (4) NVLink: A high-speed, high-bandwidth point-to-point or mesh interconnect technology used to connect multiple computing units (such as GPU to GPU, GPU to CPU) to achieve memory data sharing and task collaboration. NVLink's communication bandwidth is usually higher than that of the Peripheral Component Interconnect Express (PCIe) bus, and it has lower communication latency.

[0031] (5) NVMe-OF (NVMe over Fabrics): It is a network-oriented storage protocol encapsulation technology that allows computing modules within the same computing power cluster device, as well as computing modules between different computing power cluster devices, to access remote NVMe solid-state drives through network structures (Fabric), such as RDMA (Remote Direct Memory Access), RoCE (RDMA over Converged Ethernet), and InfiniBand. The measured access latency and bandwidth can approach the direct access performance of local NVMe solid-state drives.

[0032] (6) Fault Domain Hierarchy: This refers to classifying the computing modules in the distributed system into different logical isolation levels based on the topological distance and fault correlation between each computing module and the first computing module, with the computing module from which the write request originates (which can be called the first computing module) as the reference point. Each fault domain hierarchy corresponds to a topological distance. Computing modules within the same fault domain hierarchy have similar fault correlation with the first computing module, while different fault domain hierarchies have higher fault independence. For example, they can be divided in ascending order of topological distance as follows: Tray-level fault domain (including the first computing module), different Tray-level fault domains within the same device (including all computing modules belonging to the same computing cluster device as the first computing module), and different device-level fault domains (including all computing modules belonging to different computing cluster devices than the first computing module).

[0033] (7) Raw disk management: refers to the technology of bypassing the operating system's file system layer and directly reading, writing and managing the space of physical storage devices (such as NVMe solid-state drives) in units of sectors (the smallest physical read / write unit of traditional mechanical hard drives) or logical blocks (the smallest addressable unit presented by the storage device to the host, such as data blocks as described below). Raw disk management eliminates the metadata overhead, log overhead and lock contention of the file system, and can provide extremely low I / O (Input / Output) latency and predictable read / write performance.

[0034] (8) Direct I / O: refers to read and write operations on data that do not go through the operating system's page cache. Data can be transferred directly between the user-space buffer and the storage device, thereby avoiding the memory copy overhead and cache refresh latency between kernel mode and user mode, improving I / O throughput and reducing access latency. For example, Direct I / O can include user-space storage driver technology and direct access technology that bypasses the operating system kernel cache, such as setting the O_DIRECT flag (direct read / write flag).

[0035] (9) RDMA: is a network transmission technology that allows the computing unit or application process of one computing device to directly write or read data and memory addresses into or out of a specified memory area of ​​another computing device without the participation of the other operating system kernel and processor intervention.

[0036] (10) DMA (Direct Memory Access): is a data transfer mechanism within a computing device that allows peripherals (such as network cards, GPUs, NVMe controllers, etc.) to directly read and write data with the computing device's memory without the CPU participating in the data transfer process.

[0037] Computing clusters typically integrate dozens or even hundreds of high-performance computing units (such as GPUs and NPUs), placing extreme demands on bandwidth and latency for data delivery. If data access speed cannot match computing throughput, expensive computing resources will idle while waiting for I / O, hindering the overall efficiency of the distributed system. Therefore, computing clusters are usually equipped with massive amounts of local non-volatile storage (such as NVMe solid-state drives). Taking the GB300 supernode (a type of supernode device) as an example, it can carry 144 3.84TB NVMe solid-state drives, with a single machine's original capacity reaching approximately 552.96TB (Terabytes). If these NVMe solid-state drives, scattered in various locations, can be efficiently organized into a distributed cache pool close to the computing unit, and training, inference, and other tasks can be prioritized to access local NVMe resources, the pressure on remote shared storage can be significantly reduced, and data delivery efficiency can be significantly improved. However, many distributed systems rely on local file system cache directories, and data read and write operations involve multiple layers of transit such as VFS (Virtual File Systems), page cache, and directory entries, making it difficult to fully utilize the raw disk concurrency capabilities of a large number of NVMe solid-state drives. Furthermore, the fixed copy placement rules typically use the data center (also known as the available area of ​​the computer room), rack (the frame structure that houses the computing power cluster equipment), and computing power cluster equipment as the fault domain, lacking awareness of the computing power module, the locality within the computing power module, and the NVMe-oF access layer.

[0038] This application provides a data backup solution for distributed systems that can fully leverage the topological advantages of distributed systems. Please refer to [link to relevant documentation]. Figure 1 This is a schematic diagram of a data backup scheme provided in an embodiment of this application, as shown below. Figure 1 As shown, the scheme generally includes: Taking the source of the write request (i.e. the first computing power module) as a reference point, and based on the topological distance between the first computing power module and each computing power module in the distributed system, each computing power module in the distributed system is divided into fault domain levels with different logical isolation levels. This ensures that data replicas are distributed and stored in areas with different fault correlations, realizing topology-aware multi-level fault domain division, which effectively improves the distributed system's ability to resist regional faults (such as computing power module failure, rack power failure, and switch failure).

[0039] By combining the data volume (i.e., data volume information) of the data to be backed up (which can be referred to as the first data) with the storage status information of the computing power modules within each fault domain level, a top-down approach is adopted to progressively determine the backup allocation strategy for each fault domain level, achieving hierarchical and progressive strategy decision-making. This backup allocation strategy can precisely locate the backup target to a specific computing power module, a specific storage unit within the computing power module, a specific data block within the storage unit (each storage unit can be divided into at least one data block according to a preset granularity N, and each data block contains N data units), and even a specific sequence of data units within the data block (such as at least one consecutive and idle data unit), thereby achieving precise scheduling and load balancing of storage resources.

[0040] Furthermore, a state tree structure indexed by "maximum remaining capacity information" is constructed to manage storage space, enabling efficient space retrieval based on the state tree. For example, during backup decision-making, pruning algorithms are used to skip subtrees that do not meet capacity requirements, quickly retrieving free data blocks that do meet the capacity requirements. This optimizes the time complexity of storage space allocation to the logarithmic level, solving the bottleneck of space retrieval efficiency in large-scale distributed storage.

[0041] By implementing topology-based read optimization during the data reading phase, the system intelligently selects the nearest backup copy (such as the backup copy corresponding to the smallest topology distance) for reading based on the topology distance between the requesting computing module and the computing modules containing multiple backup copies. This proximity access mechanism significantly reduces data access latency and cross-network domain bandwidth overhead, thereby improving the overall I / O response speed of the distributed system.

[0042] Based on the physical location relationship between computing units and storage units, adaptive data transmission strategies are matched to achieve transmission path adaptation in heterogeneous scenarios. For example, in scenarios with the same computing power module, direct access technology that bypasses the operating system kernel cache or user-mode storage driver technology is used to eliminate the copy overhead between kernel mode and user mode. In scenarios with different computing power modules, on the basis of direct I / O technology, network-oriented storage protocol encapsulation technology (NVMe-OF) and remote direct memory access (RDMA) technology are further integrated to achieve high throughput and low latency remote data transmission.

[0043] On the storage unit side, a raw disk management method that bypasses the operating system's file system layer is adopted. Combined with Direct I / O technology, read and write operations are performed directly in units of logical blocks (such as data blocks), eliminating the performance loss caused by the file system layer and providing extremely low latency and predictable high-performance read and write capabilities.

[0044] Please see Figure 2 This is a schematic diagram of the architecture of a distributed system provided in an embodiment of this application. Figure 2As shown, the distributed system includes a backup management device 21 and multiple computing cluster devices (such as computing cluster devices 22a, 22b, ..., 22n). Each computing cluster device contains one or more computing modules. For example, each computing cluster device can be a GB300 supernode, which typically contains 1 to 17 computing modules (Trays) and 144 storage units (such as NVMe solid-state drives).

[0045] The backup management device 21 is a control node responsible for global resource scheduling, topology management, and metadata maintenance. In this embodiment, the backup management device 21 receives write requests from computing power modules, calculates the topological distance between the requesting computing power module and each computing power module in the distributed system based on pre-stored topology information, performs fault domain hierarchical division, and decides the backup allocation strategy for each fault domain hierarchical based on the storage status information of each computing power module in the fault domain hierarchical, thereby coordinating the flow and storage of data to be backed up among different computing power cluster devices.

[0046] The following is combined Figure 2 The data backup scheme proposed in this application is described, and the data backup scheme includes: When the first computing power module (e.g., any computing power module in the computing power cluster device 22a, as the source of data writing) needs to back up the first data (any data) to the distributed system, the first computing power module can send a write request to the backup management device 21.

[0047] In response to the write request, the backup management device 21 divides the computing modules in the distributed system into multiple logically isolated fault domain levels based on the topological distance between the first computing module and each computing module in the distributed system. Combining the data volume information of the first data and the storage status information of each computing module in each fault domain level, as well as the constructed status tree indexed by the maximum remaining capacity information, it quickly retrieves the free data blocks that meet the capacity requirements, and then decides on the backup allocation strategy for each fault domain level (which can accurately locate the specific computing module, storage unit, data block, and data unit sequence). The backup allocation strategy can then be sent to the first computing module.

[0048] After receiving the backup allocation strategy, the first computing module can adaptively match the data transmission strategy based on the backup allocation strategy and the physical location relationship between computing modules. For example, in the data backup scenario of the same computing module, Direct I / O (such as direct access technology that bypasses the operating system kernel cache or user-space storage driver technology) can be used. In the data backup scenario of cross computing modules, network-oriented storage protocol encapsulation technology (NVMe-OF) and remote direct memory access technology (RDMA) can be used to efficiently back up the first data to the storage location specified by different fault domain levels.

[0049] As can be seen, the backup data stream can bypass the backup management device 21, effectively avoiding the bandwidth bottleneck and forwarding delay under the centralized architecture. This ensures that the distributed system maintains high efficiency in physical transmission while being logically centrally managed, achieving data backup that balances disaster recovery level and transmission performance.

[0050] For example, backup management device 21 is a logical control node, responsible for control plane functions such as global topology maintenance, fault domain hierarchy division, and backup allocation strategy decision-making. Backup management device 21 can be any one or a combination of independent management nodes, strongly consistent replication groups, and embedded cluster management services to adapt to the deployment needs of distributed systems of different scales. An independent management node refers to an independent physical machine or virtual machine that deploys a dedicated metadata service process. This metadata service process is responsible for responding to write requests from computing power modules and executing subsequent computation and decision-making logic based on pre-stored topology information. A strongly consistent replication group refers to a high-availability cluster composed of multiple management nodes, maintaining metadata consistency through a consensus protocol. This high-availability cluster elects a master node to provide services. When the master node fails, the remaining nodes can be re-elected to take over the service, thereby avoiding system unavailability caused by a single point of failure. An embedded cluster management service refers to a control plane intelligent agent embedded in the computing power cluster device. This control plane intelligent agent can act as a lightweight management node, responsible for collecting and reporting topology information within the computing power cluster device, as well as receiving and executing policy instructions from the superior backup management device.

[0051] Figure 2 This is merely an illustrative representation of a possible system architecture for the technical solution of this application, and does not limit the system architecture of the technical solution of this application. That is, the technical solution of this application can also be applied to other forms of system architecture.

[0052] In this application embodiment, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations. The acquisition of personal information must be based on the knowledge or consent of the individual (or have a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws and regulations and the authorization of the personal information subject.

[0053] The data backup scheme proposed in this application will be described in detail below.

[0054] Please see Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 1 This data processing method is applied to a distributed system containing multiple computing power cluster devices, each containing at least one computing power module. The data processing method can be executed by a backup management device (such as the backup management device 21 mentioned above), and may include the following steps S31-S34: S31. Receive a write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data.

[0055] The first computing module refers to any computing module in the distributed system. When the first computing unit in the first computing module generates any data (referred to as the first data) that needs to be persisted or backed up for disaster recovery, it can initiate a write request to the backup management device. For example, in an AI training scenario, the first computing unit (such as a GPU or NPU) has completed a certain stage of model parameter calculation and needs to back up the calculated parameter data to the distributed system to prevent data loss. The first computing unit can be any computing unit in the first computing module.

[0056] The backup source address of the first data is information used to indicate the storage location of the first data at the source end. It can serve as the starting address for DMA or RDMA read operations. For example, DMA is used for reads within the same computing module, while RDMA is used for reads across computing modules. This backup source address can point to the first computing unit within the first computing module. For example, it can be a logical or physical address in the video memory and host memory of the first computing unit. A physical address refers to the actual address on a storage device (such as GPU video memory or NVMe solid-state drive) that can be directly addressed by the hardware memory controller. A logical address is an abstract address seen by the software layer or backup management device and decoupled from the physical layout. The video memory of the first computing unit is a high-speed memory soldered onto the first computing unit. The host memory of the first computing unit is the address space managed by the CPU in the computing module to which the first computing unit belongs.

[0057] The data volume information of the first data refers to the information describing the size of the storage space occupied by the first data, such as the number of bytes of the first data, for use in subsequent fault domain allocation and storage space retrieval.

[0058] S32. Based on the topological distance between the first computing power module and each computing power module in the distributed system, each computing power module in the distributed system is divided into multiple fault domain levels.

[0059] Topological distance is a metric that measures the physical or logical distance between any two computing modules in a distributed system. It typically refers to the logical interval or communication hops between them, directly reflecting the communication latency, available bandwidth, and fault correlation (or fault correlation) between them. Fault correlation refers to the probability that the other computing module will be affected when one computing module fails. The more shared infrastructure (such as power supplies, motherboards, and switches) there are, the higher the fault correlation.

[0060] For example, the backup management device can obtain the topology information of the distributed system in advance; the topology information refers to the structured data that describes the computing power cluster devices, computing power modules and their interconnection relationships in the distributed system. For example, which computing power cluster devices are included in the distributed system, which computing power modules are included in each computing power cluster device, what kind of bus (such as NVLink, PCIe, Ethernet) is used to connect the computing power cluster devices, and whether they belong to the same rack or the same data center, and which computing units and storage units are included in each computing power module, etc.

[0061] Based on this topology information, the topological distance between each computing module in multiple computing power clusters and the first computing power module can be determined; where each fault domain level corresponds to a certain topological distance. For example, the closer the topological distance, the more infrastructure the two computing power modules share, and the higher the fault correlation; they should belong to the same lower-level fault domain (e.g., this computing power module, different computing power modules in the same computing power cluster). Conversely, the farther the topological distance, the better the fault isolation; they should belong to a higher-level fault domain (e.g., different computing power clusters in different regions, different computing power clusters in the same region). The backup management device divides each computing power module in multiple computing power clusters into multiple fault domain levels according to the correspondence between topological distance and fault domain level.

[0062] In a distributed system, any computing power module is denoted as the i-th computing power module, where i is a positive integer. The following explanation uses the i-th computing power module as an example.

[0063] In one implementation, if the topology information indicates that the i-th computing module and the first computing module are the same computing module, for example, if the i-th computing module and the first computing module have the same device identifier (used to identify computing cluster devices) and module identifier (used to identify computing modules), then the topology distance between the i-th computing module and the first computing module is determined as the first distance. The first distance is the minimum value of the topology distance (e.g., it can be represented by the value "0"), indicating that their physical locations completely overlap, the communication hop count is zero, and they enjoy extremely low communication latency and extremely high fault correlation (e.g., shared power supply, motherboard, and interconnect bus).

[0064] According to the correspondence between the first distance and the first fault domain level, the i-th computing power module (i.e., the first computing power module itself) can be divided into the first fault domain level. The first fault domain level represents the "this computing power module" level fault domain. Local backup within this level can be used to achieve extremely fast write speed and extremely low recovery latency.

[0065] In another implementation, if the topology information indicates that the i-th computing module and the first computing module are not the same computing module, and the i-th computing module and the first computing module belong to the same first computing cluster device (e.g., different trays within the same supernode device), then the topology distance between the i-th computing module and the first computing module is determined as the second distance (e.g., represented by the value "1"). A second distance greater than the first distance indicates that although the i-th computing module and the first computing module are within the same computing cluster device, they are interconnected via an internal device bus (e.g., NVLink), possessing lower communication latency and motherboard and power supply level fault isolation capabilities.

[0066] The i-th computing power module can be assigned to the second fault domain level according to the correspondence between the second distance and the second fault domain level. The second fault domain level represents the fault domain level of "different computing power modules in the same computing power cluster". Within this level, computing power modules are usually interconnected through a high-speed internal bus (such as NVLink), which has low latency and achieves power supply or motherboard-level fault isolation.

[0067] In another implementation, if the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the second computing power cluster device, and the first and second computing power cluster devices are two computing power cluster devices located in the same region (e.g., the first and second computing power cluster devices are two supernode devices located in the same data center (or the same rack), then the topological distance between the i-th computing power module and the first computing power module is determined as the third distance (e.g., represented by the value "2"). A third distance greater than the second distance indicates that the i-th computing power module and the first computing power module are in different computing power cluster devices but geographically close, possessing moderate communication latency and rack or switch-level fault isolation capabilities.

[0068] The i-th computing power module can be assigned to the third fault domain level according to the correspondence between the third distance and the third fault domain level. The third fault domain level represents the fault domain of "different computing power cluster devices in the same region". This level utilizes the low network latency in the same region to achieve fault isolation between computing power cluster devices, and is suitable for dual-active or disaster recovery scenarios in the same region.

[0069] In another implementation, if the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the third computing power cluster device, and the first and third computing power cluster devices are two computing power cluster devices located in different regions (e.g., the first and third computing power cluster devices are two supernode devices located in different data centers (or different racks)), then the topological distance between the i-th computing power module and the first computing power module is determined as the fourth distance (e.g., represented by the value "3"). A fourth distance greater than the third distance indicates that the i-th computing power module and the first computing power module are located in different regions, typically interconnected via a wide area network or leased line, possessing higher communication latency and cross-regional (e.g., city-level) fault isolation capabilities.

[0070] The i-th computing power module can be assigned to the fourth fault domain level according to the correspondence between the fourth distance and the fourth fault domain level. The fourth fault domain level represents the fault domain of "different computing power cluster devices in different regions". This level can cope with large-scale faults such as regional power outages and natural disasters, providing a high level of disaster recovery capability, but correspondingly has a higher network transmission latency.

[0071] Therefore, the logic for dividing the fault domain hierarchy is shown in Table 1 below: Table 1

[0072] In one feasible implementation, the write request can carry a business importance identifier, such as an SLA (Service Level Agreement) level. After receiving the write request, the backup management device can parse the business importance identifier (such as the SLA level) carried in the write request and map different business importance identifiers to different disaster recovery strength requirements. The disaster recovery strength requirements are used to indicate the minimum fault domain level range that the backup needs to cover. For example, when the business importance of the first data is identified as high, a full fault domain backup strategy can be enabled. In this case, the computing power modules in the distributed system can be divided into four fault domain levels (i.e., first fault domain level, second fault domain level, third fault domain level, and fourth fault domain level) to achieve cross-regional disaster recovery. When the business importance of the first data is identified as medium, a partial fault domain backup strategy can be enabled. In this case, the computing power modules in the distributed system can be divided into two fault domain levels (i.e., first fault domain level and second fault domain level) or three fault domain levels (i.e., first fault domain level, second fault domain level, and third fault domain level) to achieve a balance between disaster recovery capability and backup overhead. When the business importance of the first data is identified as low, a minimum fault domain backup strategy can be enabled. In this case, the computing power modules in the distributed system can be divided into one fault domain level (i.e., first fault domain level) or two fault domain levels (i.e., first fault domain level and second fault domain level) to prioritize backup speed and storage efficiency.

[0073] This application does not limit the number of fault domain levels. Exceptions include classifying the first computing power module into one fault domain level, classifying each computing power module belonging to the same first computing power cluster device into one fault domain level, and classifying the computing power modules of other computing power cluster devices besides the first computing power cluster device into one fault domain level.

[0074] As can be seen, the embodiments of this application achieve hierarchical and precise fault isolation. By quantifying and mapping topological distances to different fault domain levels, it avoids placing all replicas under the same fault risk (such as in the same data center or rack) and prevents unnecessary overhead caused by blindly backing up across regions. Moreover, it can automatically match disaster recovery strategies of different intensities based on the business importance of the data. It uses close-range fault domains (such as different computing modules within a computing power cluster device) for fast backup and uses far-range fault domains (such as computing power cluster devices in different regions) for backup disaster recovery. This "near-fast and far-stable" strategy balances disaster recovery costs and performance requirements while ensuring data security. By dynamically identifying the topological distance between computing power modules and performing logical grouping, data backup no longer depends on fixed physical deployments but adapts to changes in topology, thereby enhancing the adaptability to complex topologies and improving the robustness and scalability of the distributed architecture in large-scale expansion and heterogeneous environments.

[0075] S33. Based on the data volume information of the first data and the storage status information of each computing module in each fault domain level, determine the backup allocation strategy for each fault domain level.

[0076] Each computing module contains at least one storage unit (such as an NVMe solid-state drive), and each storage unit is divided into at least one data block according to a preset granularity N (a positive integer). The preset granularity N refers to the basic unit size (e.g., 16 bits or 64 bytes) when logically dividing the storage space of the storage unit, which determines the precision of storage space management. The storage space of a storage unit refers to the contiguous address space that can be read and written by user-space programs after being abstracted by the underlying driver. A data block refers to the contiguous address range obtained after the storage space of a storage unit is logically divided by the preset granularity N, and is the basic logical unit for space management and allocation by the state tree.

[0077] Each data block can contain N data units. A data unit is the smallest indivisible unit of read / write operation within a data block. Its size is usually aligned with the sector size of the physical storage medium and is the basic carrier for actual data read / write operations. For example, the size of a data unit can be 1 bit or 512 bytes.

[0078] The storage status information of a computing module refers to a dynamic dataset describing the space usage and data distribution of all storage units within the module. This dataset is maintained and updated in real-time by the backup management device. The storage status information of a computing module may include the remaining capacity and allocation status information for each data block within the module. The remaining capacity information refers to the total number of data units within the data block that are not yet occupied, used to assess whether the data block meets capacity requirements. The allocation status information indicates whether each data unit within the data block is occupied; it can be a binary bitmap used to precisely monitor the micro-level usage of storage space.

[0079] For example, if the preset partitioning granularity N is 16, then a data block contains 16 data units, and the allocation status information is a 16-bit binary bitmap. For instance, the allocation status information is 0000 0000 0000 0111, where "1" indicates that the data unit is occupied, and "0" indicates that the data unit is idle. Therefore, it can be seen that the first 3 data units of this data block have been allocated, and the last 13 data units are idle; thus, the remaining capacity information of this data block is 13.

[0080] Any one of the multiple fault domain levels is represented as fault domain level j, where j is a positive integer. For example, the execution steps for determining the backup allocation strategy for fault domain level j are as follows: Step 1.1: Identify the computing module j used to back up the first data from each computing module in the fault domain level j.

[0081] This application can pinpoint the specific computing power module j used to store the first data based on the attributes of the fault domain level. The logic for determining the computing power module j, depending on the different fault domain levels, is as follows: When the fault domain level j is the first fault domain level, the computing power module j is the first computing power module in order to ensure access latency.

[0082] When the fault domain level j is the second fault domain level, the computing power module j is at least one computing power module belonging to the same first computing power cluster device as the first computing power module. At this level, other high-speed interconnected computing power modules within the same computing power cluster device can be preferentially selected for backup.

[0083] When the fault domain level j is the third fault domain level, the computing power module j is each computing power module in the fourth computing power cluster device. The fourth computing power cluster device is determined from each computing power cluster device in the same region as the first computing power cluster device to which the first computing power module belongs, according to the first decision strategy.

[0084] The first decision strategy is used to optimize transmission efficiency within the same region. For example, the first decision strategy may include at least one of the following: a fixed mapping strategy, which selects the computing cluster device for backup based on a pre-configured static mapping relationship. For instance, the attribution relationship between the first computing cluster device (source end) and one or more computing cluster devices (target end) within the same region can be pre-defined, and the backup management device directly selects the fourth computing cluster device based on this fixed attribution relationship. A load balancing strategy, which dynamically selects the computing cluster device for backup based on real-time collected operating indicators of each computing cluster device, such as CPU utilization, memory usage, I / O throughput, and network bandwidth utilization. A capacity-priority strategy, which prioritizes storage space availability as the primary decision criterion. For example, it prioritizes selecting the computing cluster device with the largest remaining storage space (or the highest percentage of remaining space) as the fourth computing cluster device.

[0085] When the fault domain level j is the fourth fault domain level, the computing power module j is each computing power module in the fifth computing power cluster device. The fifth computing power cluster device is determined from each computing power cluster device that is located in a different region from the first computing power cluster device to which the first computing power module belongs, according to the second decision strategy.

[0086] This second decision-making strategy is used to optimize cross-regional transmission efficiency. For example, the second decision-making strategy includes at least one of the following: a fixed mapping strategy, a load balancing strategy, a capacity-first strategy, a network latency strategy, and a topology-aware strategy. That is, considering the complexity of the WAN environment, in addition to the fixed mapping strategy, load balancing strategy, and capacity-first strategy, the second decision-making strategy may further include the following strategies: A network latency strategy, which involves actively probing or passively monitoring to obtain network performance indicators (such as round-trip time, jitter, and packet loss rate) between the first computing cluster device and various computing cluster devices in different regions, prioritizing the selection of computing cluster devices with low network latency and good link quality for backup. A topology-aware strategy, which involves analyzing the WAN topology to identify the routing domain paths, hop counts, and network congestion status between the first computing cluster device and various computing cluster devices in different regions, prioritizing the selection of computing cluster devices located on the same operator's backbone network or with the shortest network path, thereby avoiding cross-network transmission bottlenecks and reducing transmission costs.

[0087] Step 1.2: Based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data, determine the data block j used to back up the first data from each data block in the computing power module j.

[0088] In this application, one storage unit can correspond to one state tree. A state tree is a tree-shaped data structure used to accelerate the retrieval of free storage space. Each state tree contains at least one leaf node, and each leaf node is bound to a data block in the corresponding storage unit. Each leaf node is used to record the remaining capacity information and allocation status information of the corresponding data block.

[0089] For example, the backup management device can issue space allocation instructions to each computing cluster device. The space allocation instruction is an instruction used to trigger the computing cluster device to perform underlying storage space initialization and logical mapping. The space allocation instruction can carry a preset partition granularity N and a preset mapping function identifier to guide the computing cluster device to complete the binding from physical sectors to logical addresses.

[0090] For example, space allocation instructions can be used to instruct each computing cluster device to divide each storage unit within its own computing module into at least one data block according to a preset granularity N, and to generate a logical address for each data block based on a preset mapping function and the physical address of each data block. The preset mapping function refers to a predefined, deterministic mathematical transformation rule (such as a hash function or linear offset formula) used to map the physical address of a storage unit (such as the LBA (Logical Block Address) of an NVMe solid-state drive) to a globally unique identifier (i.e., a logical address) decoupled from its physical location.

[0091] The backup management device can receive the logical address of each data block within each storage unit returned by each computing cluster device. For example, it can report [the identifier of the computing cluster device to which the data block belongs, the identifier of the computing module to which the data block belongs, the identifier of the storage unit to which the data block belongs, and the logical address of the data block] to the backup management device. In this way, the backup management device can build a unified logical address mapping for the NVMe solid-state drives of all computing cluster devices. For example, it can establish a mapping relationship between [the identifier of the computing cluster device to which the data block belongs, the identifier of the computing module to which the data block belongs, the identifier of the storage unit to which the data block belongs, and the logical address of the data block] and the identifier of the data block. This can provide a basis for replica placement, topology distance analysis (constructing topology structure information), addressing, etc.

[0092] The backup management device can initialize at least one leaf node for each storage unit based on the logical address of at least one data block within each storage unit and a preset partition granularity N. The logical address of a data block can be bound to an initialized leaf node (to indicate the data block corresponding to the leaf node). The remaining capacity information and allocation status information recorded in an initialized leaf node can be determined based on the preset partition granularity N. For example, if the preset partition granularity N is 16, the data block corresponding to the initialized leaf node contains 16 data units, the remaining capacity information is initialized to 16 (i.e., all free), and the allocation status information is initialized to a bitmap of all "0"s. Starting from at least one leaf node of each storage unit, a preset number of child nodes (which can be set as needed, such as 2, 4, etc.) are linked to the same parent node. The maximum remaining capacity information recorded in the remaining capacity information of each child node linked to the same parent node is added to the corresponding parent node to obtain the status tree corresponding to each storage unit.

[0093] For example, the remaining capacity information can be calculated from the allocation status information. Specifically, the allocation status information is a binary bitmap of length N. By counting the number of "0"s representing idle states, the remaining capacity information can be obtained. That is, the remaining capacity information equals the preset partition granularity N minus the number of allocated data units.

[0094] Please see Figure 4 This is a schematic diagram illustrating a state tree construction process provided in an embodiment of this application. For example... Figure 4 As shown, the logical structure inside a storage unit and the state tree built for that storage unit are illustrated. The storage unit is divided into multiple data blocks (four are shown in the figure, labeled as data block 0 to data block 3). Each data block corresponds to a 32-bit binary bitmap used to record micro-allocation state information.

[0095] During the construction process, each leaf node is directly bound to a data block and carries the allocation status information of that data block. For example, the status value of data block 0 is 0x0 (hexadecimal representation), which is converted to binary as: 0000 0000 0000 00000000 0000 0000 0000; the status value of data block 1 is 0x3FFFFFFF (hexadecimal representation), which is converted to binary as: 0011 1111 1111 1111 1111 1111 1111 1111 1111; while data blocks 2 and 3 are both 0xFFFFFFFF (hexadecimal representation), which is converted to binary as: 1111 1111 1111 1111 1111 1111 1111 1111 1111 1111. The non-leaf nodes of the tree structure record the maximum value of the remaining capacity information in their child nodes. For example, the root node 0x20 (hexadecimal representation of 32) is inherited from the larger value between its child nodes 0x1E (hexadecimal representation of 30) and 0x20.

[0096] In one feasible implementation, each storage unit can correspond to multiple state trees; that is, a state tree is constructed using a batch of data blocks within each storage unit. When the capacity of a single storage unit is extremely large, dividing it into multiple independent state trees for management can effectively reduce the height of a single tree structure, thereby reducing the retrieval depth during state tree traversal and improving space allocation efficiency. For example, a large-capacity NVMe SSD with hundreds of data blocks can be divided into subtrees of 32 or 64 data blocks each, forming multiple state trees that work in parallel, ensuring retrieval speed while optimizing memory resource usage.

[0097] In one embodiment, determining the data block j for backing up the first data based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data includes: traversing the state tree corresponding to each storage unit in the computing power module j based on the data volume information of the first data. For example, before the traversal begins, the backup management device can convert the data volume information of the first data (such as the number of bytes of the first data) into the number of data units M (a positive integer) required by the first data.

[0098] During the traversal, if the remaining capacity information recorded by a traversed node does not match the data size information of the first data, all descendant nodes under that node are skipped (i.e., the subtree is pruned) until a target leaf node whose recorded remaining capacity information matches the data size information of the first data is reached. For example, if the remaining capacity information (i.e., the maximum number of unallocated data units within the node's jurisdiction) recorded by a traversed node (whether a parent node or a leaf node) is less than the number of data units M required by the first data, all descendant nodes under that node are skipped directly because that branch cannot meet the write requirement; this continues until a target leaf node whose recorded remaining capacity information is greater than or equal to the number of data units M required by the first data is reached. Here, the target leaf node refers to the lowest-level leaf node in the state tree, bound to a specific data block, and whose recorded remaining capacity information meets the write request requirements. The data block corresponding to the target leaf node is determined as data block j used to back up the first data.

[0099] As can be seen, the state tree constructed using this bottom-up approach can quickly determine whether there is sufficient space when a write request is received by comparing the data volume information of the first data with the maximum remaining capacity information recorded in the state tree. During traversal, using a pruning algorithm to skip subtrees whose remaining capacity does not meet the requirements of the first data can significantly reduce the retrieval complexity, thereby achieving fast free block retrieval and allocation, and greatly improving backup efficiency.

[0100] Multiple fault domain levels have priorities, and each fault domain level corresponds to a topological distance. The smaller the corresponding topological distance, the higher the priority of the fault domain level. For example, the priority order is: first fault domain level > second fault domain level > third fault domain level > fourth fault domain level. In a feasible implementation, if the target leaf node is not traversed (i.e., the remaining capacity information of all data blocks in fault domain level j does not match the data volume information of the first data), then the next fault domain level with a lower priority than fault domain level j is determined, and the state tree-based retrieval and pruning process is re-executed from the data blocks of the next fault domain level to determine the data block j used to back up the first data. For example, a search can be performed first at the first fault domain level (the first computing power module itself); if the target leaf node is not found, the search is downgraded to the second fault domain level (other computing power modules belonging to the same first computing power cluster device) to continue searching; if it is still not found, the search is downgraded to the third fault domain level (other computing power cluster devices in the same region) and the fourth fault domain level (other computing power cluster devices in different regions) in sequence, until a data block whose remaining capacity information matches the data volume information of the first data is found or all fault domain levels have been traversed.

[0101] For example, the first data to be backed up at the first fault domain level is called the primary replica. See also... Figure 5 This is a flowchart illustrating a replica allocation decision method provided in an embodiment of this application. The replica allocation decision method includes steps S51-S55: S51. Check if the first computing power module has storage space for the primary replica. That is, search the state tree to see if there is a target leaf node whose remaining capacity information matches the data volume information of the first data. If there is, execute S52; otherwise, execute S53.

[0102] S52. Determine that the primary replica is assigned to the first computing power module, and prioritize the use of the NVMe solid-state drive of this Tray level.

[0103] S53. Check whether each computing module belonging to the same computing power cluster device as the first computing power module has storage space for the master copy. If it does, execute S54; otherwise, execute S55.

[0104] S54. Determine the other computing modules (such as NVMe SSDs of other Trays within the same GB300 supernode) that the primary replica is assigned to the first computing cluster device.

[0105] S55. Determine which primary replica is assigned to other computing cluster devices besides the first computing cluster device, such as other GB300 nodes.

[0106] In another feasible implementation, if the target leaf node is not reached after traversing all fault domain levels, a data backup failure message can be generated. The data backup failure message is a control plane signaling used to notify the first computing power module and the backup management device that the write request cannot be fulfilled. It can carry the reason for failure (such as "insufficient free space") and trigger the exception handling logic of the upper layer application (such as alarm, retry or degradation processing).

[0107] Step 1.3: Based on the allocation status information of data block j and the data volume information of the first data, determine the backup allocation strategy of fault domain level j. The backup allocation strategy of fault domain level j is used to indicate the data unit sequence j in data block j used to back up the first data.

[0108] Each leaf node records the allocation status information of the corresponding data block, and each leaf node is bound to the logical address of the corresponding data block. In one implementation, the data unit sequence j used to back up the first data can be determined based on the data volume information of the first data and the allocation status information of data block j; the data unit sequence j contains one or more consecutive and free data units. For example, the data volume information of the first data is converted into the number M of data units required by the first data; the allocation status information (such as a bitmap) of data block j is scanned to find consecutive and free (value 0) data units; the sequence containing M consecutive free data units is extracted as the data unit sequence j.

[0109] The intra-block offset information of data unit sequence j within data block j and the logical address bound to the target leaf node are used to determine the backup destination address for fault domain level j. For example, the backup destination address can be constructed by concatenating the logical address and the intra-block offset information. The logical address is used to locate the specific storage unit (such as an NVMe solid-state drive), and the intra-block offset information is used to locate the specific read / write start position within that storage unit, that is, the starting position of the first data unit in data unit sequence j within data block j.

[0110] A backup allocation strategy for fault domain level j is generated based on the backup destination address. This backup allocation strategy may include the backup destination address and the required data length M, which instructs the first computing power module to write the first data into the corresponding data unit sequence j according to the backup destination address.

[0111] For example, when the first data is successfully backed up in data block j, the state tree corresponding to the storage unit to which data block j belongs is updated. For example, the allocation status information of the target leaf node is updated (that is, the bitmap of each data unit in the data unit sequence j is set to "1"), and the maximum remaining capacity information of all ancestor nodes of the target leaf node is updated from bottom to top to maintain the accuracy of the state tree index.

[0112] S34. Based on the backup source address of the first data, back up the first data in each fault domain level according to the backup allocation strategy of each fault domain level.

[0113] The write request is sent by the first computing unit in the first computing power module. The backup source address of the first data points to the first computing unit (such as the video memory or host memory of the first computing unit). The backup allocation strategy of fault domain level j includes the backup destination address of fault domain level j. The backup destination address of fault domain level j points to data unit sequence j and data block j.

[0114] In one embodiment, a backup instruction can be issued to the first computing unit. The backup instruction is a control signaling generated by the backup management device and sent to the first computing unit. This instruction carries at least the backup source address of the first data (pointing to the video memory or host memory of the first computing unit) and the backup destination address of the fault domain level j (containing the logical address of data block j and the intra-block offset information of data unit sequence j within data block j), used to trigger the first computing unit to perform a data backup operation. For example, the backup instruction can be used to instruct the first computing unit, based on the backup destination address of the fault domain level j and the backup source address of the first data, to establish a first transmission path from the first computing unit to the storage unit to which data block j belongs, according to a data transmission strategy, and to write the first data into the data unit sequence j in data block j through the first transmission path. The data transmission strategy refers to a set of underlying transmission mechanisms pre-selected based on the positional relationship (within the same computing module or across computing modules) between the source end (i.e., the first computing module) and the destination end (i.e., the computing module to which data block j belongs).

[0115] In one feasible implementation, after receiving a backup instruction, the first computing unit can parse the backup destination address from the instruction and establish a dedicated data channel (i.e., a first transmission path) based on the logical address in the backup destination address (which allows the storage unit to be located) and the data transmission strategy. This first transmission path connects the internal bus of the first computing unit to the controller of the storage unit to which data block j belongs. It can transmit the first data and the backup destination address to the controller of the storage unit to which data block j belongs, enabling the controller to write the first data into the data unit sequence j within data block j based on the backup destination address. For example, the controller of the storage unit to which data block j belongs can convert the logical address in the backup destination address into a physical address, thereby locating data block j using the physical address, and then using the intra-block offset information in the backup destination address to write the first data into the data unit sequence j within data block j.

[0116] For example, when the first computing unit and the storage unit to which data block j belongs belong to the same computing power module, the data transmission strategy can reduce latency by bypassing the CPU and kernel, and may include at least one of the following: direct access technology that bypasses the operating system kernel cache, user-space storage driver technology, etc. When the first computing unit and the storage unit to which data block j belongs belong to different computing power modules, the data transmission strategy can optimize network throughput and reduce cross-node latency, and may include at least one of the following: direct access technology that bypasses the operating system kernel cache, user-space storage driver technology, network-oriented storage protocol encapsulation technology (NVMe-oF), and remote direct memory access technology (RDMA).

[0117] Please see Figure 6This is a schematic diagram of a data transmission process provided in an embodiment of this application. The first computing unit belongs to computing power module-A, and data block j is located in computing power module-B. Computing power module-B can belong to the same computing power cluster device as computing power module-A (such as computing power cluster device A), or it can belong to different computing power cluster devices (such as belonging to computing power cluster device A and computing power cluster device B respectively). The first computing unit (or the first computing power module) can access NVMe solid-state drives across computing power modules as if they were local NVMe solid-state drives through Network-oriented storage protocol encapsulation technology (NVMe-oF). For example, the DataTiering service process on computing power module-B can use the server (NVMe-oF Target) of network-oriented storage protocol encapsulation technology to export the NVMe solid-state drives managed by computing power module-B. The first computing unit can use the distributed cache (Dop-cache) process on the first computing power module to connect and access the NVMe-oF Initiator (client of network-oriented storage protocol encapsulation technology).

[0118] The DataTiering service process refers to the background management program running on each computing module, used to take over local NVMe storage resources. The DataTiering service process is responsible for abstracting physical NVMe SSDs into logical storage volumes, handling read and write requests from external sources, and performing data layout management. NVMe-oF Target refers to a server-side mode that allows external nodes to access local NVMe SSDs. In this server-side mode, the DataTiering service process monitors network ports, receives and parses NVMe commands (such as read or write commands) from the NVMe-oF Initiator, and translates them into actual physical read and write operations on the local NVMe SSDs, thus making the remote NVMe SSDs appear as part of the local namespace.

[0119] The NVMe-oF Initiator refers to a client driver or user-space library deployed in the computing module (e.g., computing module-A) that initiates read / write requests. It enables the computing module to access NVMe SSDs across computing modules as if they were local NVMe SSDs. The NVMe-oF Initiator encapsulates read / write requests from the first computing unit into NVMe-oF protocol messages and sends them to a specified address. The Dop-cache process is a client agent process running on the first computing module (e.g., computing module-A), acting as the specific execution entity and network access point of the NVMe-oF Initiator. For example, it can be responsible for interacting with the first computing unit, receiving I / O requests for remote data blocks (e.g., data block j), and establishing a low-level, high-efficiency transmission channel with the DataTiering service process on the remote computing module-B via RDMA technology. During data transfer, the Dop-cache process registers the data to be written in the video memory of the first computing unit as a DMA memory region, enabling the data payload of NVMe-oF write commands to be directly transferred from the video memory of the computing unit to the remote NVMe SSD via the network, without needing to pass through the host memory. During read operations, the Dop-cache process efficiently aggregates the data stream read from the remote location and delivers it to the first computing unit.

[0120] For example, assuming the first computing unit is a GPU, data block j is located on the NVMe SSD of a remote computing module (i.e., a computing module other than the first computing module), and the selected data transfer strategy is RDMA. The backup instruction carries the backup source address (such as a logical address pointing to the GPU memory) and the backup destination address (such as containing the logical address of the remote NVMe SSD and intra-block offset information). The first computing unit can respond to the backup instruction and directly construct an RDMA write operation through the user-space driver, establishing a data transfer path from the GPU memory to the remote NVMe controller. In this process, the data does not need to pass through the CPU and operating system kernel, nor does it need to be transferred in the host memory; it can flow directly from the GPU memory to the data unit sequence j, thereby significantly reducing CPU utilization and increasing transmission bandwidth.

[0121] Therefore, this application embodiment utilizes the bottom-up aggregation and pruning algorithm of state tree to significantly reduce the complexity of free block retrieval and achieve millisecond-level fast location of massive storage space; by directly managing storage units with raw disks, direct I / O, NVMe-oF cross-computing module access, and local priority placement of primary replicas, the storage units built into multiple computing cluster devices are organized into a high-performance cache pool close to the computing unit, which can provide low transfer overhead for tasks such as training and inference; and multi-fault domain level data backup can effectively avoid the risk of single point of failure while ensuring high data reliability, and realize intelligent scheduling and load balancing of storage resources across different topology levels.

[0122] Please see Figure 7 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 The data processing method can be executed by a backup management device (such as the backup management device 21 described above), and the data processing method may include the following steps S71-S73: S71. Receive a read request sent by the second computing power module. The read request carries the identification information of the first data and the address of the requesting end.

[0123] The second computing module can be any computing module in a distributed system. The requesting address can point to the computing unit (referred to as the second computing unit) within the second computing module that initiates the read operation. For example, the requesting address can be the network reachable address or local bus identifier corresponding to the second computing unit. For instance, when a model training or inference task starts, the second computing unit can initiate a read operation when it needs to load previously backed-up data.

[0124] The identification information for the first data can be a metadata key used to uniquely identify the first data, such as a globally unique identifier for the first data, a filename hash value, or fingerprint information generated during writing. The backup allocation strategy for each fault domain level can include the backup destination address for that fault domain level. The backup destination address in each fault domain level's backup allocation strategy can be associated with the identification information of the first data. In other words, the backup management device has established and maintained a mapping relationship between the identification information of the first data and the backup destination addresses for multiple fault domain levels during previous backups.

[0125] S72. Query multiple backup destination addresses associated with the identification information of the first data, and determine the target backup destination address from the multiple backup destination addresses.

[0126] In one embodiment, the computing module corresponding to each backup destination address can be obtained, and the topological distance between the computing modules corresponding to multiple backup destination addresses and the second computing module can be determined. The backup destination address with the smallest topological distance among the multiple backup destination addresses is determined as the target backup destination address. For example, the first data backup is performed on the first computing module, other computing modules in the first computing cluster device besides the first computing module, and computing modules in computing cluster devices other than the first computing cluster device. When the second computing module and the first computing module are the same computing module, the backup management device determines that the topological distance between the first computing module and the second computing module is the smallest (local access), and selects the backup destination address at the first fault domain level as the target backup destination address to achieve nearby reading and reduce cross-node bandwidth consumption.

[0127] S73. Based on the target backup destination address and the requesting end address, read the first data into the second computing power module.

[0128] The target backup destination address points to the target data unit sequence and the target data block. For example, when the target backup destination address is backup destination address j, it points to data unit sequence j and data block j. In one embodiment, a read instruction can be sent to the second computing unit. The read instruction instructs the second computing unit to establish a second transmission path from the storage unit to which the target data block belongs to the second computing unit based on the target backup destination address and the requesting end address, according to the data transmission strategy, and to read the first data from the target data unit sequence in the target data block to the second computing unit through the second transmission path.

[0129] In one feasible implementation, after receiving a read instruction, the second computing unit can parse the target backup destination address from the read instruction and establish a dedicated data channel (i.e., a second transmission path) based on the logical address and data transmission strategy in the target backup destination address. This second transmission path connects the internal bus of the second computing unit to the controller of the storage unit to which the target data block belongs. It can transmit the target backup destination address to the controller of the storage unit to which the target data block belongs, enabling the controller to transmit the first data to the second computing unit based on the target backup destination address. For example, the controller of the storage unit to which the target data block belongs can convert the logical address in the target backup destination address into a physical address, thereby locating the target data block using the physical address, and then using the intra-block offset information in the target backup destination address to read the first data from the target data block and transmit it to the second computing unit.

[0130] For example, when the second computing unit and the storage unit to which the target data block belongs belong to the same computing power module, the data transmission strategy can reduce latency by bypassing the CPU and kernel, and may include at least one of the following: direct access technology that bypasses the operating system kernel cache, user-space storage driver technology, etc. When the second computing unit and the storage unit to which the target data block belongs belong to different computing power modules, the data transmission strategy can optimize network throughput and reduce cross-node latency, and may include at least one of the following: direct access technology that bypasses the operating system kernel cache, user-space storage driver technology, network-oriented storage protocol encapsulation technology (NVMe-oF), and remote direct memory access technology (RDMA).

[0131] To ensure high availability for read operations, this application embodiment supports an automatic replica failover mechanism. If a first data read failure is detected during the read process (e.g., target data block corruption, NVMe-oF link timeout, or connection interruption), the backup management device triggers a replica reselection process. For example, the currently failed target backup destination address is marked as unavailable, and the topology distance between it and the second computing module is recalculated based on the remaining backup destination addresses. The backup destination address with the smallest topology distance is still selected as the new target backup destination address. Based on the new target backup destination address, step S73 is re-executed to establish a second transmission path from the second nearest replica to the second computing unit, thereby achieving automatic failover without the user's awareness and ensuring the continuity of the read service.

[0132] For example, please see Figure 8 This is a schematic diagram illustrating a data reading process provided in an embodiment of this application. The local storage unit of the second computing module (containing backups of the first data) has a first priority, allowing it to read the first data locally using direct input / output technology. Only when the first data is not found in the local storage unit of the second computing module does the priority decrease to a second priority, allowing other computing modules in the same computing cluster to read the first data using a network-oriented storage protocol encapsulation technology. Only when the first data is not found in other computing modules within the same computing cluster does the priority decrease to a third priority, allowing other computing cluster devices to read the first data using a network-oriented storage protocol encapsulation technology.

[0133] This application employs a proximity-based read strategy based on topological distance, which intelligently selects the nearest replica (such as a Tray or a cluster device with similar computing power) when a read request is initiated, significantly reducing cross-node network bandwidth consumption and access latency. Simultaneously, it achieves efficient data pass-through through a data transmission strategy, avoiding the overhead of kernel and host memory transfers. Furthermore, by introducing an automatic replica failover mechanism, it can switch to the next nearest replica within milliseconds when a link or storage unit anomaly is detected. This ensures high-throughput, low-latency data supply while greatly improving the robustness and service continuity of storage access in scenarios such as distributed training and inference.

[0134] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0135] Please see Figure 9 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus includes: The receiving unit 91 is used to receive a write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data. Processing unit 92 is used to divide each computing power module in the distributed system into multiple fault domain levels based on the topological distance between the first computing power module and each computing power module in the distributed system. The processing unit 92 is also used to determine the backup allocation strategy for each fault domain level based on the data volume information of the first data and the storage status information of each computing module in each fault domain level. The processing unit 92 is also used to back up the first data in each fault domain level according to the backup source address of the first data and the backup allocation strategy of each fault domain level.

[0136] In one embodiment, the processing unit 92 divides the computing modules in the distributed system into multiple fault domain levels based on the topological distance between the first computing module and each computing module in the distributed system, including: Obtain the topology information of the distributed system; The topological distance between each computing module in a multi-computing cluster and the first computing module is determined based on topological structure information; one fault domain level corresponds to one topological distance. Based on the correspondence between topological distance and fault domain level, each computing module in multiple computing power cluster devices is divided into multiple fault domain levels.

[0137] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Processing unit 92 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on topological information, including: If the topology information indicates that the i-th computing module and the first computing module are the same computing module, then the topological distance between the i-th computing module and the first computing module is determined as the first distance; Processing unit 92 divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the first distance and the first fault domain level, the i-th computing power module unit is assigned to the first fault domain level.

[0138] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Processing unit 92 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on topological information, including: If the topology information indicates that the i-th computing power module and the first computing power module are not the same computing power module, and the i-th computing power module and the first computing power module both belong to the first computing power cluster device, then the topology distance between the i-th computing power module and the first computing power module is determined as the second distance; Processing unit 92 divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the second distance and the second fault domain level, the i-th computing power module unit is assigned to the second fault domain level.

[0139] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Processing unit 92 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on topological information, including: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the second computing power cluster device, and the first computing power cluster device and the second computing power cluster device are two computing power cluster devices located in the same region, then the topology distance between the i-th computing power module and the first computing power module is determined as the third distance. Processing unit 92 divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the third distance and the third fault domain level, the i-th computing power module unit is assigned to the third fault domain level.

[0140] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Processing unit 92 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on topological information, including: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the third computing power cluster device, and the first computing power cluster device and the third computing power cluster device are two computing power cluster devices located in different regions, then the topology distance between the i-th computing power module and the first computing power module is determined as the fourth distance. Processing unit 92 divides each computing module in multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the fourth distance and the fourth fault domain level, the i-th computing power module unit is assigned to the fourth fault domain level.

[0141] In one embodiment, each computing power module includes at least one storage unit, and each storage unit is divided into at least one data block according to a preset partitioning granularity N. The storage status information of the computing power module includes the remaining capacity information and allocation status information of each data block in the computing power module. Each data block contains N data units. The allocation status information of the data block is used to indicate whether each data unit in the data block has been allocated. Any one of the multiple fault domain levels is represented as fault domain level j, where j and N are positive integers. Based on the data volume information of the first data and the storage status information of each computing module in the fault domain level j, the processing unit 92 determines the backup allocation strategy for fault domain level j, including: From the various computing power modules at fault domain level j, determine the computing power module j used to back up the first data; Based on the remaining capacity information of each data block in computing module j and the data volume information of the first data, the data block j used to back up the first data is determined from each data block of computing module j; Based on the allocation status information of data block j and the data volume information of the first data, the backup allocation strategy of fault domain level j is determined. The backup allocation strategy of fault domain level j is used to indicate the data unit sequence j in data block j used to back up the first data. Wherein, when fault domain level j is the first fault domain level, computing module j is the first computing module; when fault domain level j is the second fault domain level, computing module j is at least one computing module belonging to the same first computing cluster device as the first computing module; when fault domain level j is the third fault domain level, computing module j is each computing module in the fourth computing cluster device, which is determined from each computing cluster device located in the same region as the first computing cluster device to which the first computing module belongs, according to a first decision strategy, the first decision strategy including at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy; when fault domain level j is the fourth fault domain level, computing module j is each computing module in the fifth computing cluster device, which is determined from each computing cluster device located in a different region from the first computing cluster device to which the first computing module belongs, according to a second decision strategy, the second decision strategy including at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy, network latency strategy, topology awareness strategy.

[0142] In one embodiment, a storage unit corresponds to a state tree, and each state tree contains at least one leaf node. A leaf node in each state tree corresponds to a data block in the corresponding storage unit. Each leaf node is used to record the remaining capacity information of the corresponding data block. Each parent node in each state tree is used to record the maximum remaining capacity information among all the remaining capacity information recorded by all child nodes under the corresponding parent node. A preset number of child nodes in each state tree are linked to the same parent node. Based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data, the processing unit 92 determines the data block j used to back up the first data from the data blocks of the computing power module j, including: Based on the data volume information of the first data, traverse the state tree corresponding to each storage unit in the computing power module j; During the traversal, if the remaining capacity information recorded by the traversed node does not match the data volume information of the first data, then skip all descendant nodes under the traversed node until the target leaf node whose recorded remaining capacity information matches the data volume information of the first data is reached. The data block corresponding to the target leaf node is determined as data block j used to back up the first data.

[0143] In one embodiment, multiple fault domain levels have priorities, each fault domain level corresponds to a topological distance, and the smaller the corresponding topological distance, the higher the priority of the fault domain level; the processing unit 92 is further configured to: If the target leaf node is not traversed, the next fault domain level with a lower priority than fault domain level j is determined, and the data block j used to back up the first data is determined from each data block of the next fault domain level.

[0144] In one embodiment, each leaf node is also used to record the allocation status information of the corresponding data block, and each leaf node is bound to the logical address of the corresponding data block. Based on the allocation status information of data block j and the data volume information of the first data, processing unit 92 determines the backup allocation strategy for fault domain level j, including: Based on the data volume information of the first data and the allocation status information of data block j, determine the data unit sequence j in data block j used to back up the first data; the data unit sequence j contains one or more consecutive and idle data units; The intra-block offset information of data unit sequence j in data block j and the logical address bound to the target leaf node are determined as the backup destination address of fault domain level j. The backup allocation strategy for fault domain level j is generated based on the backup destination address of fault domain level j.

[0145] In one embodiment, the processing unit 92 is further configured to: A space allocation instruction is issued to each computing power cluster device. The space allocation instruction is used to instruct each computing power cluster device to divide each storage unit in each computing power module into at least one data block according to the preset partitioning granularity N, and to generate the logical address of each data block based on the preset mapping function and the physical address of each data block. Receive the logical address of each data block in each storage unit returned by each computing cluster device; Based on the logical address of at least one data block in each storage unit and the preset partitioning granularity N, at least one leaf node of each storage unit is initialized. The logical address of a data block is bound to an initialized leaf node. The remaining capacity information and allocation status information recorded in an initialized leaf node are determined based on the preset partitioning granularity N. Starting from at least one leaf node of each storage unit, a preset number of child nodes are linked to the same parent node, and the maximum remaining capacity information recorded in the remaining capacity information of each child node linked to the same parent node is added to the corresponding parent node to obtain the state tree corresponding to each storage unit.

[0146] In one embodiment, the processing unit 92 is further configured to: When the first data is successfully backed up in data block j, the state tree corresponding to the storage unit to which data block j belongs is updated.

[0147] In one embodiment, each computing power module further includes at least one computing unit, the write request is sent by the first computing unit in the first computing power module, the backup source address of the first data points to the first computing unit, the backup allocation strategy of fault domain level j includes the backup destination address of fault domain level j, and the backup destination address of fault domain level j points to data unit sequence j and data block j. Processing unit 92 backs up the first data in fault domain level j according to the backup source address of the first data and the backup allocation strategy of fault domain level j, including: A backup instruction is issued to the first computing unit. The backup instruction is used to instruct the first computing unit to establish a first transmission path from the first computing unit to the storage unit to which the data block j belongs, based on the backup destination address of the fault domain level j and the backup source address of the first data, and to write the first data into the data unit sequence j in the data block j through the first transmission path. When the first computing unit and the storage unit to which data block j belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the first computing unit and the storage unit to which data block j belong belong to different computing modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

[0148] In one embodiment, the backup allocation strategy for each fault domain level includes the backup destination address for that fault domain level. The backup destination address in each fault domain level's backup allocation strategy is associated with the identification information of the first data. The receiving unit 91 is further configured to: Receive a read request sent by the second computing power module. The read request carries the identification information of the first data and the address of the requesting end. Processing unit 92 is also used for: Query multiple backup destination addresses associated with the identification information of the first data, and determine the target backup destination address from the multiple backup destination addresses; Based on the target backup destination address and the requesting end address, the first data is read into the second computing power module.

[0149] In one embodiment, the processing unit 92 determines the target backup destination address from a plurality of backup destination addresses, including: Obtain the computing power module corresponding to each backup destination address; Determine the topological distance between the computing power modules corresponding to multiple backup destination addresses and the second computing power module; The backup destination address with the smallest topological distance among multiple backup destination addresses is determined as the target backup destination address.

[0150] In one embodiment, the read request is sent by the second computing unit in the second computing power module, the requesting address points to the second computing unit, and the target backup destination address points to the target data unit sequence and the target data block; Processing unit 92 reads the first data into the second computing module based on the target backup destination address and the requesting end address, including: A read instruction is sent to the second computing unit. The read instruction is used to instruct the second computing unit to establish a second transmission path from the storage unit to which the target data block belongs to the second computing unit based on the target backup destination address and the requesting end address, and to read the first data from the target data unit sequence in the target data block to the second computing unit through the second transmission path. When the second computing unit and the storage unit to which the target data block belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the second computing unit and the storage unit to which the target data block belongs belong to different computing modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

[0151] Based on the same inventive concept, the principle and beneficial effects of the data processing device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0152] Please see Figure 10This is a schematic diagram of the structure of a computer device 100 provided in an embodiment of this application. The computer device 100 may include an input device 101, an output device 102, a processor 103, a memory 104, a network interface 105, and at least one communication bus 106. The processor 103 may be a Central Processing Unit (CPU). The processor may further include a hardware chip. The hardware chip may be an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), etc. The PLD may be a Field-Programmable Gate Array (FPGA), a Generic Array Logic (GAL), etc. Memory 104 may include volatile memory, such as random-access memory (RAM); memory 104 may also include non-volatile memory, such as flash memory, solid-state drive (SSD), etc.; memory 104 may be high-speed RAM or non-volatile memory, such as at least one disk drive. Optionally, memory 104 may also be at least one storage device located remotely from the aforementioned processor 103. Memory 104 may also include combinations of the above-mentioned types of memory. Figure 10 As shown, the memory 104, which is a computer-readable storage medium, may include an operating system, a network communication module, an object interface module, and a device control application.

[0153] Network interface 105 may include standard wired interfaces and wireless interfaces (such as Wi-Fi interfaces) for providing data communication functions; communication bus 106 is responsible for connecting various communication components; input device 101 receives instructions from object input to generate signal inputs related to object settings and function control of the terminal device. In one embodiment, input device 101 includes, but is not limited to, one or more of touch panel, physical keyboard or virtual keyboard, function keys, mouse, etc.; output device 102 is used to output data information. Output device 102 may include a display screen or other display device; processor 103 is the control center of the terminal device, connected to various parts of the entire terminal device by various interfaces and lines, and performs various functions by scheduling and running computer programs stored in memory 104.

[0154] In one embodiment, processor 103 may be used to invoke a computer program in memory to perform the following operations: Receive a write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data. Based on the topological distance between the first computing power module and each computing power module in the distributed system, each computing power module in the distributed system is divided into multiple fault domain levels. Based on the data volume information of the first data and the storage status information of each computing module in each fault domain level, a backup allocation strategy for each fault domain level is determined. Based on the backup source address of the first data, the first data is backed up in each fault domain level according to the backup allocation strategy of each fault domain level.

[0155] In one embodiment, the processor 103 divides the computing modules in the distributed system into multiple fault domain levels based on the topological distance between the first computing module and each computing module in the distributed system, including: Obtain the topology information of the distributed system; The topological distance between each computing module in a multi-computing cluster and the first computing module is determined based on topological structure information; one fault domain level corresponds to one topological distance. Based on the correspondence between topological distance and fault domain level, each computing module in multiple computing power cluster devices is divided into multiple fault domain levels.

[0156] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processor 103 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the i-th computing module and the first computing module are the same computing module, then the topological distance between the i-th computing module and the first computing module is determined as the first distance; The processor 103 divides each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the first distance and the first fault domain level, the i-th computing power module unit is assigned to the first fault domain level.

[0157] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processor 103 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the i-th computing power module and the first computing power module are not the same computing power module, and the i-th computing power module and the first computing power module both belong to the first computing power cluster device, then the topology distance between the i-th computing power module and the first computing power module is determined as the second distance; The processor 103 divides each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the second distance and the second fault domain level, the i-th computing power module unit is assigned to the second fault domain level.

[0158] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processor 103 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the second computing power cluster device, and the first computing power cluster device and the second computing power cluster device are two computing power cluster devices located in the same region, then the topology distance between the i-th computing power module and the first computing power module is determined as the third distance. The processor 103 divides each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the third distance and the third fault domain level, the i-th computing power module unit is assigned to the third fault domain level.

[0159] In one embodiment, any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; The processor 103 determines the topological distance between each computing module in the multiple computing power cluster devices and the first computing power module based on the topology information, including: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the third computing power cluster device, and the first computing power cluster device and the third computing power cluster device are two computing power cluster devices located in different regions, then the topology distance between the i-th computing power module and the first computing power module is determined as the fourth distance. The processor 103 divides each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level, including: According to the correspondence between the fourth distance and the fourth fault domain level, the i-th computing power module unit is assigned to the fourth fault domain level.

[0160] In one embodiment, each computing power module includes at least one storage unit, and each storage unit is divided into at least one data block according to a preset partitioning granularity N. The storage status information of the computing power module includes the remaining capacity information and allocation status information of each data block in the computing power module. Each data block contains N data units. The allocation status information of the data block is used to indicate whether each data unit in the data block has been allocated. Any one of the multiple fault domain levels is represented as fault domain level j, where j and N are positive integers. Based on the data volume information of the first data and the storage status information of each computing module in the fault domain level j, the processor 103 determines the backup allocation strategy for fault domain level j, including: From the various computing power modules at fault domain level j, determine the computing power module j used to back up the first data; Based on the remaining capacity information of each data block in computing module j and the data volume information of the first data, the data block j used to back up the first data is determined from each data block of computing module j; Based on the allocation status information of data block j and the data volume information of the first data, the backup allocation strategy of fault domain level j is determined. The backup allocation strategy of fault domain level j is used to indicate the data unit sequence j in data block j used to back up the first data. Wherein, when fault domain level j is the first fault domain level, computing module j is the first computing module; when fault domain level j is the second fault domain level, computing module j is at least one computing module belonging to the same first computing cluster device as the first computing module; when fault domain level j is the third fault domain level, computing module j is each computing module in the fourth computing cluster device, which is determined from each computing cluster device located in the same region as the first computing cluster device to which the first computing module belongs, according to a first decision strategy, the first decision strategy including at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy; when fault domain level j is the fourth fault domain level, computing module j is each computing module in the fifth computing cluster device, which is determined from each computing cluster device located in a different region from the first computing cluster device to which the first computing module belongs, according to a second decision strategy, the second decision strategy including at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy, network latency strategy, topology awareness strategy.

[0161] In one embodiment, a storage unit corresponds to a state tree, and each state tree contains at least one leaf node. A leaf node in each state tree corresponds to a data block in the corresponding storage unit. Each leaf node is used to record the remaining capacity information of the corresponding data block. Each parent node in each state tree is used to record the maximum remaining capacity information among all the remaining capacity information recorded by all child nodes under the corresponding parent node. A preset number of child nodes in each state tree are linked to the same parent node. Based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data, the processor 103 determines the data block j used to back up the first data from the data blocks of the computing power module j, including: Based on the data volume information of the first data, traverse the state tree corresponding to each storage unit in the computing power module j; During the traversal, if the remaining capacity information recorded by the traversed node does not match the data volume information of the first data, then skip all descendant nodes under the traversed node until the target leaf node whose recorded remaining capacity information matches the data volume information of the first data is reached. The data block corresponding to the target leaf node is determined as data block j used to back up the first data.

[0162] In one embodiment, multiple fault domain levels have priorities, each fault domain level corresponds to a topological distance, and the smaller the corresponding topological distance, the higher the priority of the fault domain level; the processor 103 is also used for: If the target leaf node is not traversed, the next fault domain level with a lower priority than fault domain level j is determined, and the data block j used to back up the first data is determined from each data block of the next fault domain level.

[0163] In one embodiment, each leaf node is also used to record the allocation status information of the corresponding data block, and each leaf node is bound to the logical address of the corresponding data block. Based on the allocation status information of data block j and the data volume information of the first data, processor 103 determines the backup allocation strategy for fault domain level j, including: Based on the data volume information of the first data and the allocation status information of data block j, determine the data unit sequence j in data block j used to back up the first data; the data unit sequence j contains one or more consecutive and idle data units; The intra-block offset information of data unit sequence j in data block j and the logical address bound to the target leaf node are determined as the backup destination address of fault domain level j. The backup allocation strategy for fault domain level j is generated based on the backup destination address of fault domain level j.

[0164] In one embodiment, the processor 103 is further configured to: A space allocation instruction is issued to each computing power cluster device. The space allocation instruction is used to instruct each computing power cluster device to divide each storage unit in each computing power module into at least one data block according to the preset partitioning granularity N, and to generate the logical address of each data block based on the preset mapping function and the physical address of each data block. Receive the logical address of each data block in each storage unit returned by each computing cluster device; Based on the logical address of at least one data block in each storage unit and the preset partitioning granularity N, at least one leaf node of each storage unit is initialized. The logical address of a data block is bound to an initialized leaf node. The remaining capacity information and allocation status information recorded in an initialized leaf node are determined based on the preset partitioning granularity N. Starting from at least one leaf node of each storage unit, a preset number of child nodes are linked to the same parent node, and the maximum remaining capacity information recorded in the remaining capacity information of each child node linked to the same parent node is added to the corresponding parent node to obtain the state tree corresponding to each storage unit.

[0165] In one embodiment, the processor 103 is further configured to: When the first data is successfully backed up in data block j, the state tree corresponding to the storage unit to which data block j belongs is updated.

[0166] In one embodiment, each computing power module further includes at least one computing unit, the write request is sent by the first computing unit in the first computing power module, the backup source address of the first data points to the first computing unit, the backup allocation strategy of fault domain level j includes the backup destination address of fault domain level j, and the backup destination address of fault domain level j points to data unit sequence j and data block j. Based on the backup source address of the first data, processor 103 backs up the first data in fault domain level j according to the backup allocation strategy of fault domain level j, including: A backup instruction is issued to the first computing unit. The backup instruction is used to instruct the first computing unit to establish a first transmission path from the first computing unit to the storage unit to which the data block j belongs, based on the backup destination address of the fault domain level j and the backup source address of the first data, and to write the first data into the data unit sequence j in the data block j through the first transmission path. When the first computing unit and the storage unit to which data block j belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the first computing unit and the storage unit to which data block j belong belong to different computing modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

[0167] In one embodiment, the backup allocation policy for each fault domain level includes the backup destination address for that fault domain level. The backup destination address in the backup allocation policy for each fault domain level is associated with the identification information of the first data. The processor 103 is further configured to: Receive a read request sent by the second computing power module. The read request carries the identification information of the first data and the address of the requesting end. Query multiple backup destination addresses associated with the identification information of the first data, and determine the target backup destination address from the multiple backup destination addresses; Based on the target backup destination address and the requesting end address, the first data is read into the second computing power module.

[0168] In one embodiment, the processor 103 determines the target backup destination address from a plurality of backup destination addresses, including: Obtain the computing power module corresponding to each backup destination address; Determine the topological distance between the computing power modules corresponding to multiple backup destination addresses and the second computing power module; The backup destination address with the smallest topological distance among multiple backup destination addresses is determined as the target backup destination address.

[0169] In one embodiment, the read request is sent by the second computing unit in the second computing power module, the requesting address points to the second computing unit, and the target backup destination address points to the target data unit sequence and the target data block; Processor 103 reads first data into the second computing module based on the target backup destination address and the requesting end address, including: A read instruction is sent to the second computing unit. The read instruction is used to instruct the second computing unit to establish a second transmission path from the storage unit to which the target data block belongs to the second computing unit based on the target backup destination address and the requesting end address, and to read the first data from the target data unit sequence in the target data block to the second computing unit through the second transmission path. When the second computing unit and the storage unit to which the target data block belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the second computing unit and the storage unit to which the target data block belongs belong to different computing modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

[0170] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, they will not be repeated here.

[0171] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0172] This application also provides a computer-readable storage medium storing a computer program for the aforementioned data processing method. The computer program includes computer instructions. When one or more processors load and execute these computer instructions, the data processing method described in the embodiments can be implemented, and will not be repeated here. The beneficial effects of using the same method will also not be repeated here. It is understood that the computer instructions can be deployed on one or more computer devices capable of communicating with each other for execution.

[0173] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0174] This application also provides a computer program product, which includes a computer program or computer instructions that, when executed by a processor, implement the above-described data processing method.

[0175] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method is applied to a distributed system, which includes multiple computing power cluster devices, each of which includes at least one computing power module. The method includes: The system receives a write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data. Based on the topological distance between the first computing power module and each computing power module in the distributed system, each computing power module in the distributed system is divided into multiple fault domain levels; Based on the data volume information of the first data and the storage status information of each computing module in each fault domain level, a backup allocation strategy for each fault domain level is determined. Based on the backup source address of the first data, the first data is backed up in each fault domain level according to the backup allocation strategy of each fault domain level; Each computing power module includes at least one storage unit, and each storage unit is divided into at least one data block according to a preset granularity N. The storage status information of the computing power module includes the remaining capacity information and allocation status information of each data block in the computing power module. Each data block contains N data units. The allocation status information of the data block is used to indicate whether each data unit in the data block has been allocated. Any one of the multiple fault domain levels is represented as fault domain level j, where j and N are positive integers. Based on the data volume information of the first data and the storage status information of each computing power module in the fault domain level j, a backup allocation strategy for the fault domain level j is determined, including: identifying a computing power module j for backing up the first data from each computing power module in the fault domain level j; identifying a data block j for backing up the first data from each data block in the computing power module j based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data; and determining a backup allocation strategy for the fault domain level j based on the allocation status information of the data block j and the data volume information of the first data. The backup allocation strategy for the fault domain level j is used to indicate the data unit sequence j in the data block j used for backing up the first data.

2. The method as described in claim 1, characterized in that, The step of dividing each computing module in the distributed system into multiple fault domain levels based on the topological distance between the first computing module and each computing module in the distributed system includes: Obtain the topology information of the distributed system; Based on the topology information, the topological distance between each computing module in the plurality of computing power cluster devices and the first computing module is determined; one fault domain level corresponds to one topological distance; Based on the correspondence between topological distance and fault domain level, each computing module in the multiple computing power cluster devices is divided into multiple fault domain levels.

3. The method as described in claim 2, characterized in that, Any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Determining the topological distance between each computing module in the plurality of computing power cluster devices and the first computing power module based on the topology information includes: If the topology information indicates that the i-th computing module and the first computing module are the same computing module, then the topological distance between the i-th computing module and the first computing module is determined as the first distance; The step of dividing each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level includes: According to the correspondence between the first distance and the first fault domain level, the i-th computing power module unit is assigned to the first fault domain level.

4. The method as described in claim 2, characterized in that, Any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Determining the topological distance between each computing module in the plurality of computing power cluster devices and the first computing power module based on the topology information includes: If the topology information indicates that the i-th computing power module and the first computing power module are not the same computing power module, and the i-th computing power module and the first computing power module both belong to the first computing power cluster device, then the topology distance between the i-th computing power module and the first computing power module is determined as the second distance; The step of dividing each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level includes: According to the correspondence between the second distance and the second fault domain level, the i-th computing power module unit is assigned to the second fault domain level.

5. The method as described in claim 2, characterized in that, Any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Determining the topological distance between each computing module in the plurality of computing power cluster devices and the first computing power module based on the topology information includes: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the second computing power cluster device, and the first computing power cluster device and the second computing power cluster device are two computing power cluster devices located in the same region, then the topological distance between the i-th computing power module and the first computing power module is determined as the third distance; The step of dividing each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level includes: According to the correspondence between the third distance and the third fault domain level, the i-th computing power module unit is assigned to the third fault domain level.

6. The method as described in claim 2, characterized in that, Any computing power module in the distributed system is represented as the i-th computing power module, where i is a positive integer; Determining the topological distance between each computing module in the plurality of computing power cluster devices and the first computing power module based on the topology information includes: If the topology information indicates that the first computing power module belongs to the first computing power cluster device, the i-th computing power module belongs to the third computing power cluster device, and the first computing power cluster device and the third computing power cluster device are two computing power cluster devices located in different regions, then the topology distance between the i-th computing power module and the first computing power module is determined as the fourth distance. The step of dividing each computing module in the multiple computing power cluster devices into multiple fault domain levels according to the correspondence between topological distance and fault domain level includes: According to the correspondence between the fourth distance and the fourth fault domain level, the i-th computing power module unit is assigned to the fourth fault domain level.

7. The method according to any one of claims 1-6, characterized in that, When the fault domain level j is the first fault domain level, the computing power module j is the first computing power module; when the fault domain level j is the second fault domain level, the computing power module j is at least one computing power module belonging to the same first computing power cluster device as the first computing power module; when the fault domain level j is the third fault domain level, the computing power module j is each computing power module in the fourth computing power cluster device, which is determined according to a first decision strategy from among the computing power cluster devices located in the same region as the first computing power cluster device to which the first computing power module belongs. The first decision strategy includes at least one of the following: fixed mapping strategy, load balancing strategy, and capacity priority strategy; when the fault domain level j is the fourth fault domain level, the computing power module j is each computing power module in the fifth computing power cluster device, and the fifth computing power cluster device is determined from each computing power cluster device located in a different region from the first computing power cluster device to which the first computing power module belongs, according to the second decision strategy, and the second decision strategy includes at least one of the following: fixed mapping strategy, load balancing strategy, capacity priority strategy, network latency strategy, and topology awareness strategy.

8. The method as described in claim 7, characterized in that, Each storage unit corresponds to a state tree, and each state tree contains at least one leaf node. A leaf node in each state tree corresponds to a data block in the corresponding storage unit. Each leaf node is used to record the remaining capacity information of the corresponding data block. Each parent node in each state tree is used to record the maximum remaining capacity information among all the remaining capacity information recorded by all child nodes under the corresponding parent node. A preset number of child nodes in each state tree are linked to the same parent node. The step of determining the data block j for backing up the first data from the data blocks of the computing power module j based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data includes: Based on the data volume information of the first data, the state tree corresponding to each storage unit in the computing power module j is traversed; During the traversal, if the remaining capacity information recorded by the traversed node does not match the data volume information of the first data, then all descendant nodes under the traversed node are skipped until the target leaf node whose recorded remaining capacity information matches the data volume information of the first data is traversed. The data block corresponding to the target leaf node is determined as data block j used to back up the first data.

9. The method as described in claim 8, characterized in that, The multiple fault domain levels have priorities. Each fault domain level corresponds to a topological distance, and the smaller the corresponding topological distance, the higher the priority of the fault domain level. The method further includes: If the target leaf node is not traversed, the next fault domain level with a lower priority than the fault domain level j is determined, and the data block j used to back up the first data is determined from each data block of the next fault domain level.

10. The method as described in claim 8, characterized in that, Each leaf node is also used to record the allocation status information of the corresponding data block, and each leaf node is bound to the logical address of the corresponding data block; The step of determining the backup allocation strategy for the fault domain level j based on the allocation status information of the data block j and the data volume information of the first data includes: Based on the data volume information of the first data and the allocation status information of the data block j, a data unit sequence j for backing up the first data is determined in the data block j; the data unit sequence j contains one or more consecutive and idle data units; The intra-block offset information of the data unit sequence j in the data block j and the logical address bound to the target leaf node are determined as the backup destination address of the fault domain level j. The backup allocation strategy for fault domain level j is generated based on the backup destination address of fault domain level j.

11. The method as described in claim 8, characterized in that, The method further includes: A space allocation instruction is issued to each of the computing power cluster devices; the space allocation instruction is used to instruct each of the computing power cluster devices to divide each storage unit in each of its computing power modules into at least one data block according to a preset granularity N, and to generate a logical address for each data block based on a preset mapping function and the physical address of each data block. Receive the logical address of each data block in each storage unit returned by each of the computing power cluster devices; Based on the logical address of at least one data block in each storage unit and the preset partitioning granularity N, at least one leaf node of each storage unit is initialized. The logical address of a data block is bound to an initialized leaf node. The remaining capacity information and allocation status information recorded in an initialized leaf node are determined based on the preset partitioning granularity N. Starting from at least one leaf node of each storage unit, a preset number of child nodes are linked to the same parent node, and the maximum remaining capacity information recorded in the remaining capacity information of each child node linked to the same parent node is added to the corresponding parent node to obtain the state tree corresponding to each storage unit.

12. The method as described in claim 8, characterized in that, The method further includes: When the first data is successfully backed up in data block j, the state tree corresponding to the storage unit to which data block j belongs is updated.

13. The method as described in claim 7, characterized in that, Each computing power module further includes at least one computing unit. The write request is sent by the first computing unit in the first computing power module. The backup source address of the first data points to the first computing unit. The backup allocation strategy of the fault domain level j includes the backup destination address of the fault domain level j. The backup destination address of the fault domain level j points to the data unit sequence j and the data block j. Based on the backup source address of the first data, the first data is backed up in the fault domain level j according to the backup allocation strategy of the fault domain level j, including: A backup instruction is issued to the first computing unit. The backup instruction is used to instruct the first computing unit to establish a first transmission path from the first computing unit to the storage unit to which the data block j belongs, based on the backup destination address of the fault domain level j and the backup source address of the first data, and to write the first data into the data unit sequence j in the data block j through the first transmission path. Wherein, when the first computing unit and the storage unit to which the data block j belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the first computing unit and the storage unit to which the data block j belongs belong to different computing power modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

14. The method as described in claim 1, characterized in that, The backup allocation strategy for each fault domain level includes the backup destination address for that fault domain level. The backup destination address in each fault domain level's backup allocation strategy is associated with the identification information of the first data. The method further includes: Receive a read request sent by the second computing power module, wherein the read request carries the identification information of the first data and the address of the requesting end; Query multiple backup destination addresses associated with the identification information of the first data, and determine the target backup destination address from the multiple backup destination addresses; Based on the target backup destination address and the requesting end address, the first data is read into the second computing power module.

15. The method as described in claim 14, characterized in that, Determining the target backup destination address from the plurality of backup destination addresses includes: Obtain the computing power module corresponding to each of the backup destination addresses; Determine the topological distance between the computing power modules corresponding to the plurality of backup destination addresses and the second computing power module; The backup destination address with the smallest topological distance among the multiple backup destination addresses is determined as the target backup destination address.

16. The method as described in claim 14, characterized in that, The read request is sent by the second computing unit in the second computing power module, the requesting address points to the second computing unit, and the target backup destination address points to the target data unit sequence and the target data block; The step of reading the first data to the second computing power module based on the target backup destination address and the requesting end address includes: A read instruction is sent to the second computing unit. The read instruction is used to instruct the second computing unit to establish a second transmission path from the storage unit to which the target data block belongs to the second computing unit based on the target backup destination address and the requesting end address, and to read the first data from the target data unit sequence in the target data block to the second computing unit through the second transmission path. Wherein, when the second computing unit and the storage unit to which the target data block belongs belong to the same computing power module, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, and user-mode storage driver technology; When the second computing unit and the storage unit to which the target data block belongs belong to different computing power modules, the data transmission strategy includes at least one of the following: direct access technology that bypasses the operating system kernel cache, user-mode storage driver technology, network-oriented storage protocol encapsulation technology, and remote direct memory access technology.

17. A data processing apparatus, characterized in that, The device is applied to a distributed system, which includes multiple computing power cluster devices, each of which includes at least one computing power module. The device comprises: The receiving unit is used to receive a write request sent by the first computing power module. The write request is used to request the backup of the first data in the distributed system. The write request carries the backup source address and data volume information of the first data. The processing unit is used to divide each computing module in the distributed system into multiple fault domain levels based on the topological distance between the first computing module and each computing module in the distributed system. The processing unit is further configured to determine the backup allocation strategy for each fault domain level based on the data volume information of the first data and the storage status information of each computing module in each fault domain level. The processing unit is further configured to back up the first data in each fault domain level according to the backup source address of the first data and the backup allocation strategy of each fault domain level; Each computing power module includes at least one storage unit, and each storage unit is divided into at least one data block according to a preset granularity N. The storage status information of the computing power module includes the remaining capacity information and allocation status information of each data block in the computing power module. Each data block contains N data units. The allocation status information of the data block is used to indicate whether each data unit in the data block has been allocated. Any one of the multiple fault domain levels is represented as fault domain level j, where j and N are positive integers. The processing unit determines a backup allocation strategy for the fault domain level j based on the data volume information of the first data and the storage status information of each computing power module in the fault domain level j. This includes: determining a computing power module j used to back up the first data from each computing power module in the fault domain level j; determining a data block j used to back up the first data from each data block in the computing power module j based on the remaining capacity information of each data block in the computing power module j and the data volume information of the first data; and determining a backup allocation strategy for the fault domain level j based on the allocation status information of the data block j and the data volume information of the first data. The backup allocation strategy for the fault domain level j is used to indicate the data unit sequence j in the data block j used to back up the first data.

18. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the data processing method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including computer instructions, which, when executed by a processor, perform the data processing method as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1 to 16.

Images