File storage method and device, electronic equipment and storage medium

Abstract

The invention provides a file storage method and device, electronic equipment and a storage medium, and relates to the technical field of Internet, in particular to the technical field of cloud computing, big data and the like. According to the scheme, a plurality of replication domains of the distributed storage cluster are obtained; wherein the replication domain is used for indicating a fault isolation unit, the replication domain comprises a plurality of logic data buckets, and the plurality of logic data buckets are distributed to different cache nodes of the distributed storage cluster; segmenting the to-be-stored file into a plurality of data blocks according to a set block size, and generating a plurality of copies of any data block; and respectively mapping the plurality of copies of any data block to different logic data buckets in the same copy domain, so that the plurality of copies are stored on a plurality of different cache nodes corresponding to the same copy domain.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, specifically to the fields of cloud computing and big data, and particularly to a file storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] As business scale continues to expand and system complexity increases, file storage architectures built on distributed cache nodes are gradually becoming an efficient implementation method. By distributing data across multiple cache nodes, not only is the system's throughput and scalability improved, but high availability and disaster recovery are also made possible. Therefore, how to store files efficiently and reliably has become a pressing technical problem to be solved in this type of distributed cache storage system. Summary of the Invention

[0003] This disclosure provides a file storage method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a file storage method is provided, comprising: Obtain multiple replication domains of a distributed storage cluster; wherein, the replication domain is used to indicate a fault isolation unit, and the replication domain includes multiple logical data buckets, which are allocated to different cache nodes of the distributed storage cluster; The file to be stored is divided into multiple data blocks according to a set block size, and multiple copies of any data block are generated; Multiple copies of any data block are mapped to different logical data buckets within the same replication domain, so that the multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

[0005] According to another aspect of this disclosure, a command processing apparatus is provided, comprising: The acquisition module is used to acquire multiple replication domains of the distributed storage cluster; wherein, the replication domain is used to indicate a fault isolation unit, and the replication domain includes multiple logical data buckets, which are allocated to different cache nodes of the distributed storage cluster; The generation module is used to divide the file to be stored into multiple data blocks according to a set block size, and generate multiple copies of any data block; The mapping module is used to map multiple copies of any data block to different logical data buckets within the same replication domain, so that the multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method proposed in one aspect of this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in the foregoing aspect of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method proposed in the foregoing aspect of this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram illustrating the principle of file storage in related technologies; Figure 2 This is a flowchart illustrating the file storage method provided in Embodiment 1 of this disclosure; Figure 3 This is a flowchart illustrating the file storage method provided in Embodiment 2 of this disclosure; Figure 4 This is a flowchart illustrating the file storage method provided in Embodiment 3 of this disclosure; Figure 5 This is a flowchart illustrating the file storage method provided in Embodiment 4 of this disclosure; Figure 6 This is a schematic diagram illustrating the principle of the file storage method provided in the embodiments of this disclosure; Figure 7 This is a schematic diagram of the structure of the file storage device provided in Embodiment 5 of this disclosure; Figure 8 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0013] In related technologies, for load balancing purposes, distributed caching systems introduce a logical data bucket layer between the mapping relationship between data and cache nodes, such as... Figure 1 As shown, during cluster deployment, N logical data buckets are initialized and evenly distributed to different cache nodes based on node capacity. After data is mapped to a bucket through consistent hashing, the cache node where the bucket is located is queried through routing information to initiate data communication. Although consistent hashing can naturally support node-level disaster recovery, it lacks explicit modeling of the topology and is difficult to cope with dynamic changes in a mixed deployment environment. While algorithms like Ceph's CRUSH can precisely control replica distribution, they heavily rely on the stability of the cluster topology. Every topology change (such as adding or deleting nodes or resource congestion) requires recalculating the placement combination of all data, resulting in large-scale data migration and high maintenance costs.

[0014] In view of at least one of the above-mentioned problems, this disclosure proposes a file storage method, apparatus, electronic device and storage medium.

[0015] The file storage method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0016] Figure 2 This is a flowchart illustrating the file storage method provided in Embodiment 1 of this disclosure.

[0017] This disclosure illustrates an example where the file storage method is configured in a file storage device. The file storage device can be applied to any electronic device so that the electronic device can perform file storage functions.

[0018] Among them, electronic devices can be any device with computing capabilities, such as computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as in-vehicle devices, mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0019] like Figure 2 As shown, the method of storing this file may include the following steps: Step 201: Obtain multiple replication domains of the distributed storage cluster.

[0020] The replication domain is used to indicate a fault isolation unit. The replication domain includes multiple logical data buckets, which are distributed to different cache nodes in the distributed storage cluster.

[0021] To achieve precise isolation of node failures, one possible approach is to obtain multiple replication domains that the system partitions during initialization or runtime. Each replication domain represents an independent fault isolation unit, and each replication domain contains a fixed number (e.g., M=3) of logical data buckets. These logical data buckets are distributed to different cache nodes in the cluster to ensure that a single point of failure does not cause the entire replication domain to fail. Here, the fault isolation unit is the basic design unit used to limit the scope of the impact of a fault, and the logical data bucket is a logical unit used to organize, partition, and manage data in a distributed storage or caching system.

[0022] Step 202: Divide the file to be stored into multiple data blocks according to the set block size, and generate multiple copies of any data block.

[0023] To improve the flexibility of data management, as one possible implementation is to divide the file to be stored into multiple data blocks according to a set block size (such as 64KB or 1MB) when a file is written, and generate multiple copies of each data block independently.

[0024] Step 203: Map multiple copies of any data block to different logical data buckets within the same replication domain, so that multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

[0025] To ensure consistency and efficient fault recovery of replicas, one possible approach is to map all replicas of the same data block to different logical data buckets within the same replication domain. These logical data buckets are then allocated to different cache nodes. This ensures that all replicas are within the same fault isolation boundary, facilitating unified management. Furthermore, if any replica is lost, it can be quickly rebuilt from other nodes within the same domain without cross-domain coordination.

[0026] The file storage method of this disclosure constructs multiple replication domains, each serving as an independent fault isolation unit containing multiple logical data buckets. These logical data buckets are allocated to different cache nodes in a distributed storage cluster. Furthermore, the file to be stored is divided into multiple data blocks according to a set block size, and multiple replicas are generated for each data block. These replicas are mapped to different logical data buckets within the same replication domain, ensuring that each replica is actually stored on multiple different cache nodes corresponding to that replication domain. Therefore, when any single cache node fails, only the portion of data it carries is affected, while the remaining replicas of the same data block remain intact on other healthy nodes within the same domain, achieving efficient node-level disaster recovery. Simultaneously, since the replica distribution is strictly limited to the same replication domain, the additional overhead and management complexity caused by cross-domain redundancy are avoided, and parallel access by clients based on replication domain information is facilitated, significantly improving the system's performance stability and robustness in a hybrid deployment environment.

[0027] To clearly illustrate how multiple copies of any data block are mapped to different logical data buckets within the same replication domain in the above embodiments, so that multiple copies are stored on multiple different cache nodes corresponding to the same replication domain, this disclosure proposes another file storage method.

[0028] Figure 3 This is a flowchart illustrating the file storage method provided in Embodiment 2 of this disclosure.

[0029] like Figure 3 As shown, the method for storing this file may include the following steps: Step 301: Obtain multiple replication domains of the distributed storage cluster.

[0030] The replication domain is used to indicate a fault isolation unit. The replication domain includes multiple logical data buckets, which are distributed to different cache nodes in the distributed storage cluster.

[0031] To achieve node-level fault isolation, one possible approach is to divide multiple logical data buckets in the distributed storage cluster into multiple fault isolation units (each corresponding to a replication domain), and based on the capacity information of each cache node, allocate the logical data buckets within each replication domain to different cache nodes.

[0032] As an example, multiple logical data buckets in a distributed storage cluster are divided into multiple fault isolation units; each fault isolation unit corresponds to a replication domain; based on the capacity information of each cache node in the distributed storage cluster, the logical data buckets in each replication domain are allocated to different cache nodes.

[0033] In other words, the existing logical data buckets in the entire cluster (such as data units generated by hash sharding or range sharding) are divided into several fault isolation units. Each fault isolation unit is logically independent and corresponds to a replication domain. That is, all data buckets in the unit share the same set of replication policies and fault boundaries. The system further combines the capacity information of each cache node (such as remaining storage space, load status, etc.) to make physical deployment decisions for the logical data buckets in each replication domain. That is, each logical data bucket in the same replication domain is allocated to different cache nodes. This ensures that even if any cache node fails or goes down, any logical data bucket in the replication domain can still be accessed normally through its replicas on other nodes, thus effectively tolerating node-level failures. At the same time, since the allocation process fully considers the capacity of the nodes, it avoids writing a large amount of data to high-load or low-capacity nodes, achieving balanced utilization of cluster storage resources and improving overall performance and scalability.

[0034] To ensure continuous disaster recovery and resource balance in a dynamically changing distributed environment, one possible approach is to dynamically trigger the reallocation of logical data buckets when topology changes (such as node additions, offline events, or network domain adjustments) and / or capacity changes (such as storage resource releases, load surges, or disk failures) are detected in the distributed storage cluster. For example, when the topology changes, the division of fault isolation units is reassessed based on the updated topology information, and the affiliation and distribution of logical data buckets within each replication domain are adjusted accordingly to ensure that replicas always meet the disaster recovery constraints across nodes and within the same domain. Alternatively, when the node capacity status changes, logical data buckets are migrated or rebalanced based on the latest capacity information to avoid local overload or resource idleness.

[0035] Step 302: Divide the file to be stored into multiple data blocks according to the set block size, and generate multiple copies of any data block.

[0036] Step 303: In response to the first number of multiple replicas of any data block being equal to the second number of multiple logical data buckets within the same replication domain, the multiple replicas are mapped to the multiple logical data buckets within the same replication domain.

[0037] Each replica corresponds one-to-one with a logical data bucket within the same replication domain.

[0038] It's important to understand that the size M of the replication domain (i.e., the number of logical data buckets it contains) defines the maximum number of replicas that can be supported within that domain. To achieve matching between replicas and disaster recovery resources, as a possible implementation, when the number of replicas configured by the user equals the number of logical data buckets within the replication domain (e.g., replica count = 3, replication domain contains 3 buckets), the system maps each replica one-to-one to each logical bucket in the replication domain, ensuring even distribution of replicas, no redundant empty buckets, and maximizing storage resource utilization. Simultaneously, because each logical data bucket carries a unique replica, load balancing is satisfied, and this mapping method ensures that replicas cover the entire fault isolation unit. When any node fails, all other replicas come from different nodes, resulting in the shortest recovery path and high reconstruction efficiency.

[0039] Step 304: In response to the first quantity being less than the second quantity, select a target logical data bucket equal to the first quantity from multiple logical data buckets within the same replication domain, and map the multiple replicas to the target logical data bucket respectively.

[0040] To improve the flexibility of replica distribution, as a possible implementation, when the number of replicas specified by the user is less than the total number of logical buckets in the replication domain (e.g., number of replicas = 2, replication domain contains 3 buckets), the system dynamically selects a target logical data bucket from the replication domain that is equal to the number of replicas for mapping, and maps multiple replicas to the target logical data buckets respectively.

[0041] It should be noted that the execution process of step 302 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further. Steps 303 and 304 are two parallel execution methods. In actual execution, one of them is selected.

[0042] In summary, when the number of replicas of any data block equals the number of logical data buckets within the replication domain, the system can achieve a one-to-one mapping between replicas and buckets, ensuring that each replica resides on a different cache node, thereby maximizing the disaster recovery capability of the replication domain. When the number of replicas is less than the number of buckets, the system dynamically selects a target logical data bucket from the replication domain with an equal number of replicas for mapping, ensuring that replicas are distributed across nodes while avoiding resource waste or mapping conflicts. Thus, user-configured replicas are strictly constrained to different cache nodes within the same replication domain, maintaining disaster recovery capability while balancing storage efficiency and deployment flexibility, significantly improving the system's reliability, scalability, and fault recovery efficiency in a hybrid deployment environment.

[0043] To clearly illustrate how multiple copies of any data block are generated in the above embodiments, this disclosure proposes another file storage method.

[0044] Figure 4This is a flowchart illustrating the file storage method provided in Embodiment 3 of this disclosure.

[0045] like Figure 4 As shown, the method for storing this file may include the following steps: Step 401: Obtain multiple replication domains of the distributed storage cluster.

[0046] The replication domain is used to indicate a fault isolation unit. The replication domain includes multiple logical data buckets, which are distributed to different cache nodes in the distributed storage cluster.

[0047] Step 402: Divide the file to be stored into multiple data blocks according to the set block size.

[0048] Step 403: Obtain the copy configuration information associated with the file to be stored.

[0049] To achieve differentiated storage services, one possible approach is to obtain the copy configuration information associated with the file to be stored; this copy configuration information may originate from user specifications or the business context.

[0050] Step 404: Determine the number of replicas for any data block based on the replica configuration information.

[0051] Furthermore, after obtaining the replica configuration, the number of replicas generated for each data block of the file to be stored is dynamically determined based on the replica configuration information. In other words, even in the same cluster, different files can have different redundancy levels, or even different data blocks of the same file can correspond to different numbers of replicas. Thus, the configuration can be differentiated according to the user's actual needs, avoiding resource waste or insufficient reliability.

[0052] Step 405: Based on the number of replicas, copy any data block to obtain multiple replicas of any data block.

[0053] To achieve redundant data generation consistent with the configuration, one possible approach is to perform a certain number of copy operations on the original data blocks based on a predetermined number of copies, thereby generating a specified number of copies.

[0054] Step 406: Map multiple copies of any data block to different logical data buckets within the same replication domain, so that multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

[0055] It should be noted that the execution processes of steps 401, 402 and 406 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit these processes and will not elaborate further.

[0056] In summary, by obtaining the replica configuration information associated with the file to be stored, the system can perceive the user's specific needs. Then, based on the replica configuration information, it determines the number of replicas that should be generated for each data block, and accurately replicates the data block according to this number to generate the corresponding number of replicas. This provides a data foundation for subsequently mapping the data to different logical data buckets within the same replication domain and distributing it to multiple cache nodes. This enables the system to simultaneously support the storage needs of highly reliable critical business and low-cost temporary data in the same cluster, ensuring disaster recovery capabilities while avoiding unnecessary storage overhead, and significantly improving resource utilization efficiency and service flexibility.

[0057] Based on any of the above embodiments, when a persistent failure occurs in the target cache node, the system recovers the missing data from replicas of other healthy nodes within the same replication domain, based on the replication domain to which the logical data bucket it carries belongs, and writes the reconstructed data into an idle logical data bucket to achieve fault recovery. The following section combines... Figure 5 The embodiments are described in detail.

[0058] Figure 5 This is a flowchart illustrating the file storage method provided in Embodiment 4 of this disclosure.

[0059] like Figure 5 As shown, the method for storing this file may include the following steps: Step 501: Obtain multiple replication domains of the distributed storage cluster.

[0060] The replication domain is used to indicate a fault isolation unit. The replication domain includes multiple logical data buckets, which are distributed to different cache nodes in the distributed storage cluster.

[0061] Step 502: Divide the file to be stored into multiple data blocks according to the set block size, and generate multiple copies of any data block.

[0062] Step 503: Map multiple copies of any data block to different logical data buckets within the same replication domain, so that multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

[0063] Step 504: In response to the detection of a fault in the target cache node and its failure to recover within a set time period, determine the logical data buckets carried on the target cache node, and determine the first target replication domain to which any of the carried logical data buckets belongs.

[0064] To achieve accurate and efficient fault location and recovery scope definition, as a possible implementation method, when the system confirms that a cache node has failed and cannot be recovered within a set time period, it can identify all logical data buckets carried on the cache node and further determine the replication domain (i.e., the first target replication domain) to which each bucket belongs. It should be noted that since each replication domain is a fault isolation unit, cross-domain coordination overhead can be avoided, significantly improving recovery efficiency.

[0065] Step 505: For any logical data bucket carried, obtain valid replica data from the logical data buckets carried by other cache nodes within the first target replication domain, excluding the target cache node.

[0066] Because the replication domain design ensures that replicas are distributed across different nodes, even if one node fails completely, the remaining replicas remain fully available. Furthermore, valid replica data is read from the corresponding logical data buckets on other healthy nodes within the first target replication domain.

[0067] Step 506: Based on the valid replica data, reconstruct the missing data in any logical data bucket of the host, and write the reconstructed data into the idle logical data buckets of other cache nodes.

[0068] To achieve automatic data repair in fault scenarios, the system uses the obtained valid copies to reconstruct all missing data in the original bucket and writes the reconstruction results into the idle logical data buckets (i.e., pre-allocated but not full or released buckets) of other nodes in the cluster.

[0069] To achieve low-latency data reading, one possible approach is to concurrently access multiple replica nodes within the replication domain when reading data blocks, and terminate the remaining requests immediately after any node returns valid data.

[0070] As an example, in response to reading any data block from the distributed storage cluster, a list of cache nodes containing each replica in the second target replication domain mapped to the data block is obtained; based on the list of cache nodes, a read request is initiated to the cache nodes according to a preset multi-replica access strategy; in response to detecting that any cache node returns valid data, the remaining read requests are terminated and the valid data is used.

[0071] In other words, when reading any data block from the distributed storage cluster, the system obtains a list of cache nodes distributed among all replicas within the second target replication domain to which the data block belongs; then, according to the preset multi-replica access strategy (such as concurrent read, priority read), the system initiates read requests to these nodes simultaneously or in sequence; once it detects that any one of the cache nodes has successfully returned valid data, the system immediately terminates the remaining ongoing read requests and returns the valid data to the client.

[0072] In addition, it should be noted that when the system detects a timeout or error return (such as network interruption, service unresponsiveness, data verification failure, etc.) in a cache node during the reading process, it will immediately mark the node as abnormal. This allows for the rapid identification and isolation of potential slow or faulty nodes, preventing abnormal nodes from affecting the response performance of read requests or causing repeated errors. In subsequent read operations on data blocks within the same replication domain, the system will automatically skip the abnormal node and only initiate access from the cache nodes where other healthy replicas are located, thereby ensuring low latency and high success rate of read operations.

[0073] It should be noted that the execution process of steps 501 to 503 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0074] In summary, when a target cache node fails and fails to recover within a set time period, all logical data buckets carried on that node can be located, and the first target replication domain to which each bucket belongs can be determined. Since logical data buckets within the same replication domain are pre-allocated to multiple different cache nodes, valid replica data can be obtained from the corresponding logical buckets carried by other healthy nodes within that replication domain. Based on these valid replicas, the system reconstructs all data lost in the original bucket due to node failure and writes the reconstruction results to idle logical data buckets on other nodes. This ensures that the data recovery process is strictly confined within the fault isolation unit, eliminating the need for cross-domain coordination and significantly reducing recovery latency and network overhead. Simultaneously, using pre-allocated idle buckets for reconstruction avoids metadata bloat and scheduling complexity caused by dynamically creating storage units, thus achieving efficient and automated fault self-healing capabilities while ensuring data persistence and consistency.

[0075] In any of the above embodiments, as Figure 6 As shown, the file storage method of this disclosure embodiment specifically includes the following steps: (1) Define the replication domain size M. At this time, the cluster file can support a maximum of M replicas for caching (the default is 3, which can meet most caching scenarios). Each replica of the file is evenly mapped to different data buckets within the same replication domain for data reading and writing. For example, when M=3, BG1, BG2, and BG3 form a replication group. The three replicas of the 3-replica file are placed in {BG1, BG2, BG3} respectively, and the two replicas of the 2-replica file are placed in {BG1, BG2} or {BG2, BG3} or {BG3, BG1} respectively. (2) The cluster divides all data buckets into multiple replication domains according to the step size M. During initialization and data balancing, it ensures that buckets in the same replication domain are eventually placed on different cache nodes, so that the buckets in the replication domain meet the node-level tolerance. As the cache node topology changes (adding and deleting cache nodes, the mixed disk being occupied by other business processes, etc.), the buckets in the replication domain are flexibly migrated between multiple nodes by the Master scheduler. The placement combination is not fixed, and the migration process does not need to consider the existing multiple replicas of files. (3) When a bucket is emptied (e.g., a business node fails directly and cannot be added back), the cached data is migrated from other buckets in the same replication domain to repair it. In other words, in this distributed caching architecture, when a business node suffers a permanent failure (e.g., hardware damage that cannot be recovered) and its multiple data buckets are emptied, the system can use the replica data on other healthy nodes in the same replication domain to automatically trigger the cache repair process. Considering that the number of buckets on a single node is between 500 and 800 and the number of nodes is less than 100, the number of other nodes (scatterwidth) participating in data recovery when a single node fails can easily reach the number of nodes in the entire cluster, thus accelerating the fault recovery. (4) Based on the preset replication domain, the client side implements multi-replica failover and backup to maintain cache performance and reduce the impact of slow and bad nodes in a hybrid environment. Specifically, when the client initiates a read request for a data block, it first obtains the list of cache nodes corresponding to all replicas of the data block in the second target replication domain from the distributed storage cluster. Then, according to the preset multi-replica access strategy (such as concurrent access, priority round-robin, or latency awareness), it sends read requests to multiple cache nodes simultaneously or sequentially. Once any cache node successfully returns valid data, the client immediately terminates the remaining unfinished read requests and uses the valid data for subsequent processing. This mechanism not only effectively avoids the problem of slow nodes or bad nodes caused by hardware heterogeneity, resource contention, or network jitter in a hybrid environment, but also significantly improves the response speed of cache access and the overall stability of the system by quickly switching to healthy replicas through failover, thereby optimizing cache performance while ensuring data consistency.

[0076] With the above Figures 1 to 6 Corresponding to the file storage method provided in the embodiments, this disclosure also provides a file storage device. Since the file storage device provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 6 The file storage method provided in the embodiments corresponds to the file storage method provided in the embodiments of this disclosure, and therefore the implementation of the file storage method is also applicable to the file storage device provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0077] Figure 7 This is a schematic diagram of the structure of the file storage device provided in Embodiment 5 of this disclosure.

[0078] like Figure 7 As shown, the file storage device 700 includes: an acquisition module 710, a generation module 720, and a mapping module 730.

[0079] The acquisition module 710 is used to acquire multiple replication domains of the distributed storage cluster; wherein, a replication domain is used to indicate a fault isolation unit, and the replication domain includes multiple logical data buckets, which are allocated to different cache nodes of the distributed storage cluster; the generation module 720 is used to divide the file to be stored into multiple data blocks according to a set block size, and generate multiple copies of any data block; the mapping module 730 is used to map the multiple copies of any data block to different logical data buckets within the same replication domain, so that the multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

[0080] As one possible implementation, the mapping module 730 is used to map the multiple replicas to the multiple logical data buckets in the same replication domain in response to a first number of multiple replicas of any data block being equal to a second number of multiple logical data buckets in the same replication domain; wherein, the multiple replicas correspond one-to-one with the multiple logical data buckets in the same replication domain; in response to a first number being less than a second number, a target logical data bucket equal to the first number is selected from the multiple logical data buckets in the same replication domain, and the multiple replicas are mapped to the target logical data buckets.

[0081] As one possible implementation, the file storage device 700 also includes a determination module.

[0082] The determination module is used to divide multiple logical data buckets in the distributed storage cluster into multiple fault isolation units; each fault isolation unit corresponds to a replication domain; and the logical data buckets in each replication domain are allocated to different cache nodes according to the capacity information of each cache node in the distributed storage cluster.

[0083] As one possible implementation, the generation module 720 is used to obtain the copy configuration information associated with the file to be stored; determine the number of copies of any data block based on the copy configuration information; and copy any data block based on the number of copies to obtain multiple copies of any data block.

[0084] As one possible implementation, the file storage device 700 also includes an update module.

[0085] The update module is used to respond to the detection of a topology change in the cache nodes of the distributed storage cluster, and to reallocate the logical data buckets in each replication domain based on the updated topology information; and / or, in response to the detection of a capacity change in the cache nodes of the distributed storage cluster, to reallocate the logical data buckets in each replication domain based on the updated capacity information.

[0086] As one possible implementation, the file storage device 700 also includes a reconstruction module.

[0087] The reconstruction module is used to respond to the detection of a fault in the target cache node and its failure to recover within a set period of time, determine the logical data buckets carried on the target cache node, and determine the first target replication domain to which any of the carried logical data buckets belongs; for any carried logical data bucket, obtain valid replica data from the logical data buckets carried by other cache nodes in the first target replication domain other than the target cache node; based on the valid replica data, reconstruct the missing data in any carried logical data bucket, and write the reconstructed data into the idle logical data buckets of other cache nodes.

[0088] As one possible implementation, the file storage device 700 also includes a reading module.

[0089] The reading module is configured to: respond to reading any data block from the distributed storage cluster; obtain a list of cache nodes containing each replica in the second target replication domain mapped to the data block; initiate a read request to the cache nodes based on the cache node list and according to a preset multi-replica access strategy; and terminate the remaining read requests and use the valid data in response to detecting that any cache node returns valid data.

[0090] As one possible implementation, the file storage device 700 also includes a tagging module.

[0091] The marking module is used to mark any cache node as abnormal in response to the detection of any cache node timeout or return error; in subsequent read operations, cache nodes marked as abnormal are skipped until the abnormal cache node recovers to normal.

[0092] The file storage device of this disclosure constructs multiple replication domains, each serving as an independent fault isolation unit containing multiple logical data buckets. These logical data buckets are allocated to different cache nodes in a distributed storage cluster. Furthermore, the file to be stored is divided into multiple data blocks according to a set block size, and multiple replicas are generated for each data block. These replicas are mapped to different logical data buckets within the same replication domain, ensuring that each replica is actually stored on multiple different cache nodes corresponding to that replication domain. Therefore, when any single cache node fails, only the portion of data it carries is affected, while the remaining replicas of the same data block remain intact on other healthy nodes within the same domain, achieving efficient node-level disaster recovery. Simultaneously, since the replica distribution is strictly limited to the same replication domain, the additional overhead and management complexity caused by cross-domain redundancy are avoided, and parallel access by clients based on replication domain information is facilitated, significantly improving the system's performance stability and robustness in a hybrid deployment environment.

[0093] To implement the above embodiments, this disclosure also provides an electronic device, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the file storage method proposed in any of the above embodiments of this disclosure.

[0094] To implement the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the file storage method proposed in any of the above embodiments of this disclosure.

[0095] To implement the above embodiments, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the file storage method proposed in any of the above embodiments of this disclosure.

[0096] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0097] Figure 8A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 802 or a computer program loaded from storage unit 808 into RAM (Random Access Memory) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0099] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0100] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the file storage method described above. For example, in some embodiments, the file storage method described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the file storage method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the file storage method described above by any other suitable means (e.g., by means of firmware).

[0101] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0105] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0106] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0107] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0108] According to the technical solution of this disclosure, multiple replication domains are constructed, each replication domain serving as an independent fault isolation unit, containing multiple logical data buckets. These logical data buckets are allocated to different cache nodes in the distributed storage cluster. Furthermore, the file to be stored is divided into multiple data blocks according to a set block size, and multiple replicas are generated for any data block. These replicas are mapped to different logical data buckets within the same replication domain, so that each replica is actually stored on multiple different cache nodes corresponding to that replication domain. Therefore, when any single-point cache node fails, only the portion of data it carries is affected, while the remaining replicas of the same data block remain intact on other healthy nodes within the same domain, thus achieving efficient node-level disaster recovery. Simultaneously, since the replica distribution is strictly limited to within the same replication domain, the additional overhead and management complexity caused by cross-domain redundancy are avoided, and it facilitates parallel access by clients based on replication domain information, significantly improving the system's performance stability and robustness in a hybrid deployment environment.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution proposed in this disclosure can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A file storage method, characterized in that, include: Obtain multiple replication domains of the distributed storage cluster; wherein, the replication domain is used to indicate a fault isolation unit, and the replication domain includes multiple logical data buckets, which are allocated to different cache nodes of the distributed storage cluster; The file to be stored is divided into multiple data blocks according to a set block size, and multiple copies of any data block are generated; Multiple copies of any data block are mapped to different logical data buckets within the same replication domain, so that the multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

2. The method according to claim 1, wherein, The step of mapping multiple replicas of any data block to different logical data buckets within the same replication domain, so that the multiple replicas are stored on multiple different cache nodes corresponding to the same replication domain, includes: In response to the first number of replicas of any data block being equal to the second number of logical data buckets within the same replication domain, the replicas are mapped to the logical data buckets within the same replication domain, respectively; wherein, the replicas correspond one-to-one with the logical data buckets within the same replication domain. In response to the first quantity being less than the second quantity, a target logical data bucket equal to the first quantity is selected from multiple logical data buckets within the same replication domain, and the multiple replicas are respectively mapped to the target logical data bucket.

3. The method according to claim 1, wherein, The multiple logical data buckets in the multiple replication domains are determined using the following steps: The distributed storage cluster is divided into multiple logical data buckets into multiple fault isolation units; each fault isolation unit corresponds to a replication domain. Based on the capacity information of each cache node in the distributed storage cluster, the logical data buckets in each replication domain are allocated to different cache nodes.

4. The method according to claim 1, wherein, The generation of multiple copies of any data block includes: Obtain the copy configuration information associated with the file to be stored; Based on the replica configuration information, determine the number of replicas for any data block; Based on the number of replicas, any one of the data blocks is copied to obtain multiple replicas of the any one of the data blocks.

5. The method according to claim 1, wherein, The method further includes: In response to the detection of a topology change in the cache nodes of the distributed storage cluster, the logical data buckets within each replication domain are reallocated based on the updated topology information; and / or, In response to the detection of a capacity change in the cache nodes of the distributed storage cluster, the logical data buckets within each replication domain are reallocated based on the updated capacity information.

6. The method according to claim 1, wherein, The method further includes: In response to the detection of a fault in the target cache node and its failure to recover within a set period of time, the logical data buckets carried on the target cache node are determined, and the first target replication domain to which any of the carried logical data buckets belongs is determined; For any logical data bucket, obtain valid replica data from the logical data buckets carried by other cache nodes within the first target replication domain, excluding the target cache node; Based on the valid replica data, the missing data in any of the logical data buckets of the carrier is reconstructed, and the reconstructed data is written into the idle logical data buckets of the other cache nodes.

7. The method according to any one of claims 1-6, wherein, The method further includes: In response to reading any data block from the distributed storage cluster, obtain a list of cache nodes where each replica in the second target replication domain mapped to the any data block is located; Based on the cache node list, a read request is initiated to the cache node according to a preset multi-replica access strategy; In response to the detection that any of the cache nodes has returned valid data, the remaining read requests are terminated and the valid data is used.

8. The method according to claim 7, wherein, The method further includes: In response to detecting a timeout or error return from any of the cache nodes, mark any of the cache nodes as abnormal; In subsequent read operations, skip cache nodes marked as abnormal until the abnormal cache nodes return to normal.

9. A file storage device, characterized in that, include: The acquisition module is used to acquire multiple replication domains of the distributed storage cluster; wherein, the replication domain is used to indicate a fault isolation unit, and the replication domain includes multiple logical data buckets, which are allocated to different cache nodes of the distributed storage cluster; The generation module is used to divide the file to be stored into multiple data blocks according to a set block size, and generate multiple copies of any data block; The mapping module is used to map multiple copies of any data block to different logical data buckets within the same replication domain, so that the multiple copies are stored on multiple different cache nodes corresponding to the same replication domain.

10. An electronic device, wherein, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images