Method and system for realizing multi-copy file storage based on fragmented storage

Abstract

The invention discloses a method and a system for realizing multi-copy file storage based on fragmented storage, which realize an efficient and flexible file storage management scheme by organizing storage nodes through a chain structure, selecting a target node through load balancing, configuring a redundancy number and adjusting the number of execution files and supporting dynamic extension. According to the method, the chain structure organizes the storage nodes, and the architecture design of the system is simplified. And selecting a target node in load balancing, and dynamically adjusting a storage strategy according to the real-time state of the node by supporting multiple algorithms. And a flexible and reliable configuration mechanism is provided for file storage by configuring the redundancy number and executing the step of adjusting the file number. The system can configure different redundancy numbers according to user requirements so as to realize different levels of file storage reliability. The invention provides a file storage management process which is efficient, flexible and easy to expand, the file storage requirements in different scenes can be met, and the overall performance and maintainability of the system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-copy file storage, and in particular to a method and system for implementing multi-copy file storage based on shard storage. Background Art

[0002] With the development of product library systems, more and more companies are beginning to use file storage subsystems to manage and store various product files. Existing file storage systems, such as HDFS, FastDFS, Amazon S3, and FTP file servers, have been widely used in the field of file storage. Systems such as HDFS have the advantages of supporting high fault tolerance, high throughput, and strong scalability, and are particularly suitable for large file storage. However, these systems perform poorly in scenarios with small file storage and high real-time requirements, and their complex architectural design leads to high deployment and maintenance costs. At the same time, open source distributed file systems such as HDFS and FastDFS usually introduce complex master-slave architectures and network communication modules, which brings additional complexity and potential maintenance risks to the overall architecture of the product library system.

[0003] In addition to the storage subsystem, the product library system also includes multiple other subsystems. Therefore, the architecture of the file storage subsystem should not affect the design and maintenance of the entire product library system. The existing file storage solutions are difficult to flexibly adapt to the product library system's needs for storage subsystems due to their complex architecture, especially in terms of ensuring high reliability and dynamic scalability of files. Therefore, there is an urgent need for a file storage subsystem with a simple architecture, easy maintenance, and high reliability to meet the actual needs of the product library system. Summary of the invention

[0004] The present application provides a method and system for implementing multi-copy file storage based on shard storage, aiming to solve the problem that existing file storage solutions are difficult to flexibly adapt to the requirements of the product library system for the storage subsystem due to their complex architecture, especially in terms of ensuring high reliability and dynamic scalability of files.

[0005] In a first aspect, a method for implementing multi-copy file storage based on shard storage, the method comprising:

[0006] Organize multiple storage nodes into a chain structure and maintain the readable and writable status of each node;

[0007] The redundancy number of the configuration file defines the number of storage copies of the file;

[0008] Select the target storage node according to the load balancing algorithm, and upload the file to multiple target storage nodes in parallel;

[0009] Check the actual number of copies of the file. If the number of copies does not meet the redundancy configuration, adjust the number of files.

[0010] When a new storage node is added, the chain structure is updated and file storage requests are redistributed according to the load balancing algorithm.

[0011] In the above scheme, optionally, the load balancing algorithm includes a random selection algorithm, a free space proportion algorithm and a free space absolute value algorithm.

[0012] In the above scheme, optionally, the method further includes: after the file is uploaded, the system verifies the stored copy to confirm that the copy has been uploaded successfully.

[0013] In the above scheme, optionally, the method includes a dynamic expansion step, when a new storage node is added, the chain structure is updated, and the storage requests are redistributed according to the load balancing algorithm.

[0014] In the above solution, optionally, the file quantity adjustment operation includes selecting a target storage node to perform file compensation or deleting redundant copies.

[0015] In a second aspect, a system for implementing multi-copy file storage based on shard storage is provided, the system comprising:

[0016] Multiple storage nodes are organized in a chain structure, and each storage node contains a storage disk and a node status detection module;

[0017] A load balancing module, used to select one or more target storage nodes for file storage according to a preset load balancing algorithm;

[0018] A redundancy configuration module, used to upload files to multiple target storage nodes in parallel according to the redundancy number configured by the user;

[0019] The file self-balancing module is used to detect the number of file copies on each storage node and perform file quantity adjustment operations when the actual number of file copies does not match the configured redundancy number;

[0020] The dynamic expansion module is used to update the chain structure when a new storage node is added and allocate new file storage requests according to the load balancing algorithm.

[0021] In the above solution, optionally, the chain structure includes a default processing node for processing file storage requests that are not matched to a storage node.

[0022] In the above solution, optionally, the load balancing module supports a random selection algorithm, a free space proportion algorithm and a free space absolute value algorithm.

[0023] In the above solution, optionally, the redundant configuration module verifies the upload result during the file upload process to ensure that all copies are uploaded successfully.

[0024] In the above solution, optionally, the file self-balancing module is used to select a target storage node to perform a file quantity adjustment operation according to a difference in the number of file copies.

[0025] Compared with the prior art, this application has at least the following beneficial effects:

[0026] Based on further analysis and research on the existing technical problems, this application recognizes that the existing file storage solutions are difficult to flexibly adapt to the needs of the product library system for the storage subsystem due to the complex architecture, especially in terms of ensuring the high reliability and dynamic scalability of the files. The decentralized chain structure is adopted to organize the disk nodes in series, without the need for master-slave nodes, and through a variety of load balancing algorithms and redundant configuration mechanisms, the high-reliability storage of files and the dynamic expansion of disk capacity are realized. The system checks and adjusts the number of files on each disk through a regular self-balancing mechanism (Rebalance Mechanism) to ensure that the number of file copies is consistent with the configured redundancy number, thereby improving the self-healing ability and overall reliability of the system. Compared with the existing file storage system, the chain structure adopted by the present invention not only simplifies the system architecture and reduces the network communication overhead, but also realizes the high-reliability storage and dynamic expansion capability of the files through redundant configuration and load balancing algorithms. The system supports a variety of load balancing algorithms, such as random selection, free space ratio, free space absolute value and other algorithms, and can select appropriate storage strategies according to different application scenarios. In addition, the system also provides standardized file upload and download APIs, supports user-defined extensions, and realizes non-intrusive function extensions.

[0027] The present application organizes storage nodes through a chain structure, selects target nodes through load balancing, configures redundancy, performs file quantity adjustment, and supports dynamic expansion, thereby realizing an efficient and flexible file storage management solution. The chain structure of the method organizes storage nodes and simplifies the system architecture design. In traditional distributed storage systems, nodes usually use a master-slave architecture for communication and synchronization, while the chain structure adopts a decentralized design and does not rely on the control of the master node. Each storage node maintains its own state and links to the next node, thereby reducing the communication overhead between nodes and improving the scalability of the system. Secondly, the step of load balancing to select the target node can dynamically adjust the storage strategy according to the real-time state of the node by supporting multiple algorithms, such as random selection, free space ratio and free space absolute value algorithm. Unlike the fixed node selection method, this method can flexibly select the target storage node according to the actual situation, thereby improving the balance and success rate of file storage. The steps of configuring the redundancy number and performing the file quantity adjustment provide a flexible redundant configuration mechanism for file storage. The system can configure different redundancy numbers according to user needs to achieve different levels of file storage reliability. In actual operation, the file quantity adjustment operation can detect and correct the situation where the number of copies deviates from the redundant configuration, and maintain the reasonable distribution of file copies. The design that supports dynamic expansion enables the system to automatically update the chain structure and reallocate storage requests when adding new storage nodes, without downtime for maintenance, which improves the flexibility and scalability of the system.

[0028] This application designs an efficient, flexible and easy-to-expand file storage management process that can meet the file storage needs in different scenarios and improve the overall performance and maintainability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a method for implementing multi-copy file storage based on shard storage provided in one embodiment of the present application;

[0030] Figure 2 A schematic diagram of a simplified process of uploading a shard storage system provided in one embodiment of the present application;

[0031] Figure 3 A schematic diagram of the architecture design of a shard storage system provided for one embodiment of the present application;

[0032] Figure 4 A schematic diagram of a chain organization of storage capabilities provided for one embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] In one embodiment, Figure 1 As shown, a method for implementing multi-copy file storage based on shard storage is provided, comprising the following steps:

[0035] Organize multiple storage nodes into a chain structure and maintain the readable and writable status of each node;

[0036] The redundancy number of the configuration file defines the number of storage copies of the file;

[0037] Select the target storage node according to the load balancing algorithm, and upload the file to multiple target storage nodes in parallel;

[0038] Check the actual number of copies of the file. If the number of copies does not meet the redundancy configuration, adjust the number of files.

[0039] When a new storage node is added, the chain structure is updated and file storage requests are redistributed according to the load balancing algorithm.

[0040] This embodiment organizes storage nodes through a chain structure, selects target nodes through load balancing, configures redundancy numbers, performs file quantity adjustment, and supports dynamic expansion, thereby realizing an efficient and flexible file storage management solution. The chain structure of the method organizes storage nodes and simplifies the system architecture design. In traditional distributed storage systems, nodes usually use a master-slave architecture for communication and synchronization, while the chain structure adopts a decentralized design and does not rely on the control of the master node. Each storage node maintains its own state and links to the next node, thereby reducing the communication overhead between nodes and improving the scalability of the system. Secondly, the step of load balancing to select the target node can dynamically adjust the storage strategy according to the real-time state of the node by supporting multiple algorithms, such as random selection, free space ratio and free space absolute value algorithm. Different from the fixed node selection method, this method can flexibly select the target storage node according to the actual situation, thereby improving the balance and success rate of file storage. The steps of configuring the redundancy number and performing the file quantity adjustment provide a flexible redundant configuration mechanism for file storage. The system can configure different redundancy numbers according to user needs to achieve different levels of file storage reliability. In actual operation, the file quantity adjustment operation can detect and correct the situation where the number of copies deviates from the redundant configuration, and maintain the reasonable distribution of file copies. The design that supports dynamic expansion enables the system to automatically update the chain structure and reallocate storage requests when adding new storage nodes, without downtime for maintenance, which improves the flexibility and scalability of the system.

[0041] In this embodiment, the load balancing algorithm includes a random selection algorithm, a free space ratio algorithm, and a free space absolute value algorithm.

[0042] The file quantity adjustment operation of this embodiment includes randomly selecting redundant copies on the target storage node for deletion when it is detected that the number of file copies is more than the configured redundancy number. This design provides an efficient space management mechanism for the file storage system, which can optimize the utilization of storage resources without affecting the data integrity. First, the redundant copy deletion operation can effectively release storage space. In traditional file storage systems, there is often a lack of dynamic copy quantity management mechanism. Once the number of file copies exceeds the configured redundancy number, the system cannot automatically clean up the redundant copies, resulting in storage space being occupied by unnecessary data. Especially in the scenario of high redundancy configuration, the storage node may have insufficient disk space due to storing too many copies, affecting the overall performance of the system. By randomly selecting redundant copies for deletion, the system can reclaim storage space in time and reduce unnecessary storage overhead. Secondly, this design can improve the load balancing effect of storage nodes. During the file storage process, the load of nodes is often unbalanced, and some nodes may be frequently selected as target storage nodes, thereby storing too many file copies, resulting in excessive load. By randomly selecting redundant copies for deletion, the system can dynamically adjust the storage load of nodes so that the storage pressure of each node tends to be balanced. Compared with fixed deletion of copies of a certain node, this random deletion strategy is more flexible and adaptable. In addition, randomly selecting and deleting redundant copies can also improve the system's fault tolerance and data recovery capabilities. In some cases, if the system always selects copies of the same node for deletion, it may cause insufficient number of file copies on some nodes, increasing the risk of data loss. The random deletion strategy avoids this risk by evenly distributing deletion operations, improving data security and system stability.

[0043] The file quantity adjustment operation described in this embodiment provides a flexible and efficient space management mechanism by randomly selecting redundant copies for deletion, which can optimize storage resource utilization, improve node load balancing effects, and reduce the risk of data loss.

[0044] In this embodiment, the method further includes: after the file is uploaded, the system verifies the stored copy to confirm that the copy has been uploaded successfully.

[0045] In the dynamic expansion step of this embodiment, when a new storage node is detected to be added to the chain structure, the system automatically updates the chain table and reallocates the file storage request according to the load balancing algorithm. This feature brings excellent scalability and flexibility to the system, enabling it to cope with dynamic changes in storage requirements.

[0046] The design of the dynamic expansion step solves the problem of complex node expansion process in traditional file storage systems. In many existing distributed storage systems, adding new storage nodes often requires manual configuration and synchronization of node information, which is complex and error-prone. The dynamic expansion step simplifies the node expansion process by automatically detecting new nodes and updating the chain structure, reduces the system maintenance workload, and improves the operability and reliability of the system. Updating the chain structure and reallocating file storage requests enables the new nodes to immediately participate in file storage work, effectively improving the storage efficiency and processing capacity of the system. Through this design, the system can quickly utilize the storage resources of the new nodes and avoid the problem of idle node resources after expansion. At the same time, reallocating file storage requests helps to balance the node load, reduce the storage pressure of a single node, and improve the overall performance of the system. The combination of the dynamic expansion step and the load balancing algorithm enables the system to intelligently select the target storage node according to the status and load of the new nodes. Compared with the traditional fixed allocation strategy, the load balancing algorithm can dynamically adjust the storage request allocation according to the real-time status of the node, avoiding the situation where the new nodes are overused or completely idle after joining. This intelligent allocation strategy further optimizes the resource utilization efficiency of the system and improves the response speed of storage requests.

[0047] The dynamic expansion step described in this embodiment realizes flexible expansion of the system by automatically detecting newly added nodes, updating the chain structure and reallocating storage requests, simplifies the node expansion process, and improves the storage efficiency and load balancing capability of the system.

[0048] In this embodiment, the method includes a dynamic expansion step. When a new storage node is added, the chain structure is updated and storage requests are redistributed according to a load balancing algorithm.

[0049] The standardized file upload and download API design of this embodiment supports users to implement custom plug-ins and configure them in a chain structure to achieve functional expansion and personalized storage needs. This design brings extremely high flexibility and scalability to the file storage system, and can adapt to the special needs of different users. The standardized API provides a unified interface specification, which simplifies the interaction between users and the system. Traditional file storage systems often have inconsistent interfaces, and users need to do a lot of adaptation work when integrating and using them, which increases development and maintenance costs. Through the standardized API design, users only need to call according to the unified interface specification, without paying attention to the specific implementation details within the system, which significantly reduces the difficulty of use. Supporting custom plug-in extension functions enables the system to flexibly adapt to the needs of different scenarios. In actual applications, different users may have large differences in their needs for file storage. For example, some users need specific encrypted upload functions, or need to select storage nodes according to custom rules. Through the plug-in mechanism, users can achieve personalized functional expansion without modifying the core code of the system, which improves the adaptability and maintainability of the system. The chain structure configuration method of the custom plug-in can further simplify the plug-in management process. Users can directly define the loading order and associated nodes of plug-ins through XML configuration files without writing additional code, thereby improving the operability of the system. At the same time, this chain configuration method is consistent with the chain structure design of the system, ensuring the efficiency and consistency of the plug-in calling process.

[0050] The standardized API design and custom plug-in support described in this embodiment provide a flexible and efficient expansion method for the system, which can meet the diverse needs of users and improve the flexibility and usability of the system.

[0051] In this embodiment, the file quantity adjustment operation includes selecting a target storage node to perform file compensation or deleting redundant copies.

[0052] The XML configuration file design described in this embodiment provides a structured and easy-to-manage configuration method for the system, improves the maintainability and flexibility of the system, and simplifies the configuration process of storage nodes and algorithm parameters.

[0053] In one embodiment, a system for implementing multi-copy file storage based on shard storage is provided, including:

[0054] Multiple storage nodes are organized in a chain structure, and each storage node contains a storage disk and a node status detection module;

[0055] A load balancing module, used to select one or more target storage nodes for file storage according to a preset load balancing algorithm;

[0056] A redundancy configuration module, used to upload files to multiple target storage nodes in parallel according to the redundancy number configured by the user;

[0057] The file self-balancing module is used to detect the number of file copies on each storage node and perform file quantity adjustment operations when the actual number of file copies does not match the configured redundancy number;

[0058] The dynamic expansion module is used to update the chain structure when a new storage node is added and allocate new file storage requests according to the load balancing algorithm.

[0059] The system of this embodiment adopts a chain structure to organize multiple storage nodes, and solves the problems of complexity, insufficient reliability and poor scalability of file storage architecture in the prior art through the collaborative work of load balancing module, redundant configuration module, file self-balancing module and dynamic expansion module. The chain structure organizes storage nodes, avoiding the master-slave architecture and node communication overhead commonly seen in traditional distributed storage systems. The master-slave architecture usually requires frequent communication between nodes to synchronize the state, resulting in complex system architecture and difficult maintenance. Through the chain structure, each node only needs to maintain its own state and link with the next node, reducing the communication requirements between nodes and simplifying the system architecture design. The chain structure can also dynamically adjust the node order as needed, thereby improving the flexibility of the system. The load balancing module selects the target storage node for file storage through a random selection algorithm, a free space ratio algorithm and a free space absolute value algorithm, effectively dispersing the load of storage requests and avoiding the performance degradation of some nodes due to overload. Compared with the simple polling algorithm or fixed node selection strategy in the prior art, the load balancing module of the present invention can make intelligent selections according to the real-time load conditions of the nodes, thereby improving the efficiency and balance of file storage. The redundant configuration module uploads files to multiple storage nodes in parallel according to the redundancy number configured by the user, thus realizing multi-copy storage of files. Compared with the existing single-copy storage method, multi-copy storage can significantly improve the storage reliability of files and avoid data loss caused by single node failure. The file self-balancing module can regularly check the number of file copies on each storage node, and adjust it according to the actual number of copies and the configured redundancy number to ensure that the number of file copies is consistent with the redundant configuration. The prior art lacks an effective file self-balancing mechanism, which often leads to an imbalance in the number of file copies. The present invention avoids the problem of unbalanced node storage by automatically adjusting the number of files.

[0060] In this embodiment, the system achieves architecture simplification, load balancing, high storage reliability and dynamic expansion through chain structure organization and multi-module collaboration, overcoming the existing defects in the background technology.

[0061] In this embodiment, the chain structure includes a default processing node for processing file storage requests that are not matched to a storage node.

[0062] This embodiment provides a chain structure including a default processing node, which is used to process file storage requests that are not matched to a suitable storage node. The introduction of the default processing node adds a "bottom line" mechanism to the file storage system, that is, when the storage node in the chain structure cannot meet the file storage conditions, the system can automatically forward the file storage request to the default processing node for processing. The default processing node can improve the robustness of the system. In a traditional file storage system, if all storage nodes cannot meet the storage conditions (such as insufficient disk space, node unavailability, etc.), the system will fail to store, resulting in the inability to respond to user requests. By introducing the default processing node, the system can provide a backup processing path in this case to avoid the loss or error of the storage request. The design of the default processing node improves the flexibility of the system. The administrator can dynamically adjust the configuration of the default processing node according to the system load and node status, for example, select a node with a lower load as the default processing node during peak hours to optimize system performance. At the same time, the default processing node can also be configured as a high-performance storage node to ensure storage efficiency under special circumstances. The introduction of the default processing node simplifies the design of the chain structure. In practical applications, the chain structure may be adjusted due to changes in node status, which increases the complexity of system design and maintenance. By adding default processing nodes, the chain structure can reduce the complexity of node status judgment when processing storage requests and improve processing efficiency.

[0063] The default processing node design of this embodiment improves the robustness, flexibility and processing efficiency of the file storage system, and makes up for the insufficient processing capability of the traditional chain structure under abnormal circumstances.

[0064] In this embodiment, the load balancing module supports a random selection algorithm, a free space proportion algorithm, and a free space absolute value algorithm.

[0065] This embodiment provides a load balancing module that supports a random selection algorithm, a free space ratio algorithm, and a free space absolute value algorithm. By supporting multiple load balancing algorithms, the system can select a suitable algorithm according to the actual application scenario, thereby improving the efficiency and performance of file storage. The application of the random selection algorithm can evenly distribute storage requests in simple scenarios, avoiding the situation where the load is concentrated on a specific node. For an environment where the node status is relatively balanced, the random selection algorithm can make decisions quickly, reduce computing overhead, and improve the response speed of the system. The free space ratio algorithm calculates the ratio of the free space of the node to the total space and selects the node with the highest ratio for file storage. This algorithm is suitable for the situation where the free space of the node is unevenly distributed, and effectively avoids the storage failure of the node due to insufficient space. Compared with the traditional fixed node selection strategy, the free space ratio algorithm can dynamically adjust the decision according to the real-time status of the node, thereby improving the success rate of storage. The free space absolute value algorithm selects the target storage node based on the actual remaining space size of the node, and is suitable for scenarios where large-capacity nodes need to be used first. This algorithm can make full use of the storage resources of large-capacity nodes in the system, extend the service life of small-capacity nodes, and optimize the resource allocation of the system.

[0066] The load balancing module described in this embodiment supports multiple algorithm selections and can be dynamically adjusted according to system load conditions and node status to meet the needs of different application scenarios, thereby improving the balance and efficiency of file storage.

[0067] In this embodiment, the redundant configuration module verifies the upload result during the file upload process to ensure that all copies are uploaded successfully.

[0068] The redundant configuration module of this embodiment verifies the upload result during the file upload process to ensure that all copies are successfully stored on the target storage node. This feature provides a data consistency guarantee mechanism for the system, which can effectively solve the problem of missing copies caused by network fluctuations, node failures or storage anomalies in traditional file storage systems. In the actual operation of the file storage system, the file upload operation may be affected by various factors, such as network interruption, node failure, disk write failure, etc. If there is no verification mechanism, the system may not be able to detect the failure of the file copy upload in time, resulting in the loss or incompleteness of some file copies, thereby affecting the integrity and reliability of the data. By verifying the upload result, the system can verify the consistency of each stored copy after the file upload operation is completed to ensure that all copies have been successfully stored on the selected target node. The verification process of the redundant configuration module can be implemented in a variety of ways, such as hash verification, file size verification or storage path verification. These verification methods can quickly detect abnormal situations that occur during the upload process and trigger the error handling mechanism in time. Through this design, the system can immediately perform compensation operations when the file upload fails, such as reselecting the target node for uploading or adjusting the storage strategy according to the load balancing algorithm, thereby improving the success rate of file upload. The file upload verification mechanism can also improve the system's fault tolerance. In a large-scale distributed storage environment, communication and data synchronization between nodes are often affected by latency and network jitter, which may cause some nodes to fail to receive the uploaded file data in time. By verifying the upload results, the system can effectively detect these abnormal situations and take corresponding recovery measures to ensure the integrity and consistency of the file data.

[0069] The redundant configuration module described in this embodiment provides a data consistency guarantee mechanism by verifying the upload result, which can effectively detect and handle abnormal situations occurring during the file upload process, thereby improving the reliability and fault tolerance of the file storage system.

[0070] In this embodiment, the file self-balancing module is used to select a target storage node to perform a file quantity adjustment operation according to the difference in the number of file copies.

[0071] In this embodiment, the file self-balancing module selects the target storage node to perform the file quantity adjustment operation according to the difference in the number of file copies. This module design provides the system with a capability to dynamically adjust the number of file copies, avoiding the problem of uneven distribution of file copies between storage nodes.

[0072] In traditional file storage systems, the number of file copies is usually pre-configured, but in actual operation, due to factors such as changes in node status and file access frequency, the actual number of file copies often deviates from the preset redundancy number. For example, when a storage node exits due to a failure, the file copy stored on the node will be lost, resulting in an insufficient number of file copies. When a node is mistakenly selected as a target node multiple times, the number of file copies may exceed the preset redundancy number. This uneven distribution of the number of copies may cause some nodes to be overloaded, affecting the overall performance of the system.

[0073] The file self-balancing module can dynamically keep the number of file copies consistent with the configured redundancy number by regularly detecting the number of file copies on each storage node and adjusting the number of files based on the detection results. Specifically, when it is detected that the number of file copies is insufficient, the module will select a suitable target node to supplement the missing copies; when it is detected that the number of copies is more than the configuration, it will randomly select nodes to delete the excess copies. Through this adjustment operation of more deletion and less supplementation, the file self-balancing module can achieve a dynamic balance of file copies among storage nodes.

[0074] In addition, the file self-balancing module can also adapt to changes in system load. For example, during peak hours, some nodes may experience response delays due to high load. By adjusting the distribution of file copies, the module can effectively alleviate the load pressure of the nodes.

[0075] In one embodiment, Figure 2-Figure 4 As shown, this embodiment designs a storage subsystem suitable for product libraries with high file reliability and scalable disk capacity. Users can define the number of file copies. When the number of file copies actually stored in the system is not equal to the defined number of file copies, the balancing mechanism will be automatically triggered to perform differential processing on the actual number of copies and the defined number of copies. This balancing mechanism not only assists the system in improving the high reliability of files, but also provides a basic capability for the scalability of disk capacity. The specific invention points are as follows:

[0076] Use a chain structure to organize different disk information and disk load balancing algorithms for program reading;

[0077] The system will periodically determine the disk status by judging whether the disk is readable or writable. Unavailable disks will not participate in file upload and download processing.

[0078] When uploading and downloading files, you can configure a variety of disk load balancing algorithms: random, free space percentage, and free space absolute value algorithm. Customers can use different algorithms for different scenarios.

[0079] Provides a configurable file copy number mechanism to upload files to multiple physical disks in parallel to ensure high file reliability;

[0080] A balancing mechanism for the number of file copies is provided. When the actual number of file copies is not equal to the defined number of file copies, the system automatically performs difference elimination to maximize the number of file copies to the configured number of copies.

[0081] like Figure 2 The schematic diagram of the simple upload process of the shard storage system provided is as follows: the user uploads the file, and the business layer receives the user's upload file request and starts processing;

[0082] The virtual file service processes upload requests at the business layer, shielding the processing differences between different storage subsystems;

[0083] File cache processing, if the cache is turned on, the pre-processing of writing files to the cache is performed;

[0084] File failure retry processing. If this capability is enabled, the pre-processing such as the number of retries and the retry time will be processed;

[0085] The shard storage subsystem processes and lists disks in the list to be uploaded based on algorithms such as disk availability and disk load;

[0086] If parallel processing is enabled, the thread pool is enabled to traverse the list to be uploaded and upload the files to different physical disks in parallel. Otherwise, the files are uploaded serially.

[0087] If the number of successful uploads returned by the physical disk is greater than the configured minimum successful upload threshold, the file is considered to have been uploaded successfully, and the file is subsequently cached. If the upload fails, return to step 4 for processing.

[0088] like Figure 3 The provided shard storage system architecture design diagram shows that when the product library service receives a request to upload a file, the file is processed by the shard storage system and stored in different physical disks in the form of multiple copies. Figure 3 As shown in the figure, each file is stored in two copies on different physical disks. If the available space of the entire physical disk storage is about to be exhausted, the user can add more free physical disks to alleviate the problem.

[0089] like Figure 4 A schematic diagram of the chain organization of storage capacity is provided, and an embodiment is explained as follows:

[0090] Cache capability: Provides accelerated file access capability. The cache directory can be mounted to high-performance disks such as high-speed SSDs.

[0091] Failure retry capability: When a file operation fails, basic retry capability is provided. The retry capability includes the number of retries and the retry time.

[0092] File sharding storage capability: The core implementation of the file sharding storage subsystem, where files are stored in multiple copies;

[0093] Default processing capability: The default processing node can return a null value or report an error.

[0094] Different storage capabilities are organized in the form of a linked list, and the default processing capability is appended to the tail node. When operating on a file, it is processed sequentially from the head node. When the file exists, it is returned according to the processing logic of the matching node. When the file does not exist, it is processed according to the default processing node.

[0095] Compared with the prior art, the advantages of this embodiment are:

[0096] Simple storage architecture: Each disk node does not require network communication and does not have the master-slave concept, which greatly simplifies the architecture of the storage subsystem. Each node only maintains its own status for business use. Although there is a certain risk of data inconsistency, this risk can be eliminated through the file balancing mechanism.

[0097] Easy-to-maintain configuration definition method: The advantage of using a chain structure to organize disk information is that it is convenient for the program to implement the chain of responsibility model. In addition, this configuration method is more readable, and users can easily identify which disks in the system are organized and interacted in what way;

[0098] Extensible plug-in mechanism: It provides standard file upload and download interface capabilities. Users only need to implement the standard interface and configure the custom functions in the configuration file in a chain structure without intrusion on the file system itself.

[0099] System self-healing capability: The system will periodically detect the actual number of copies of files on the disk in the background. If it is found that there is a difference with the configured number of copies, it will randomly delete the disk if there are more copies. If there are fewer copies, it will use the configured load balancing algorithm to select a suitable disk for file compensation.

[0100] The advantage of this embodiment is that it provides high reliability and expandable capacity for the product library storage subsystem through a simple architecture. The protection points are as follows:

[0101] A definition and load method of multi-disk storage, characterized by organizing various disk information in a chain structure, and often ending with a default processing node;

[0102] Highly reliable file storage capability, characterized by uploading files to multiple disks at the same time through a load balancing algorithm (random, free space percentage, free space absolute value), and the node where each disk is located is stateless and does not require communication;

[0103] A balancing mechanism for the number of file copies, characterized in that the system will ensure that the number of file copies actually stored on the disk always reaches the configured number of copies within a certain period, and its strategy is to delete more and add less;

[0104] A capability that allows disk capacity to be dynamically expanded. Its characteristic is that the idle disk information is configured in the configuration file in a chain structure. When uploading files, the system will select the idlest disk to save the file through its disk load balancing algorithm.

[0105] The specific implementation content of each module can be found in the above-mentioned method for implementing multi-copy file storage based on shard storage, which will not be repeated here.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for implementing multi-copy file storage based on shard storage, characterized in that: The method comprises: Organize multiple storage nodes into a chain structure and maintain the readable and writable status of each node; The redundancy number of the configuration file defines the number of storage copies of the file; Select the target storage node according to the load balancing algorithm, and upload the file to multiple target storage nodes in parallel; Check the actual number of copies of the file. If the number of copies does not meet the redundancy configuration, adjust the number of files. When a new storage node is added, the chain structure is updated and file storage requests are redistributed according to the load balancing algorithm.

2. The method according to claim 1, characterized in that The load balancing algorithm includes a random selection algorithm, a free space ratio algorithm and a free space absolute value algorithm.

3. The method according to claim 1, characterized in that The method further includes: after the file is uploaded, the system verifies the stored copy to confirm that the copy is uploaded successfully.

4. The method according to claim 1, characterized in that: The method includes dynamic expansion capability. When new storage nodes are added, the chain structure is updated and storage requests are redistributed according to a load balancing algorithm.

5. The method according to claim 1, characterized in that The file quantity adjustment operation includes selecting a target storage node to perform file compensation or deleting redundant copies.

6. A system for implementing multi-copy file storage based on shard storage, characterized in that: The system comprises: Multiple storage nodes are organized in a chain structure, and each storage node contains a storage disk and a node status detection module; A load balancing module, used to select one or more target storage nodes for file storage according to a preset load balancing algorithm; A redundancy configuration module, used to upload files to multiple target storage nodes in parallel according to the redundancy number configured by the user; The file self-balancing module is used to detect the number of file copies on each storage node and perform file quantity adjustment operations when the actual number of file copies does not match the configured redundancy number; The dynamic expansion module is used to update the chain structure when a new storage node is added and allocate new file storage requests according to the load balancing algorithm.

7. The system according to claim 6, characterized in that The chain structure includes a default processing node for processing file storage requests that are not matched to a storage node.

8. The system according to claim 6, characterized in that The load balancing module supports a random selection algorithm, a free space ratio algorithm, and a free space absolute value algorithm.

9. The system according to claim 6, characterized in that The redundant configuration module verifies the upload result during the file upload process to ensure that all copies are uploaded successfully.

10. The system according to claim 6, characterized in that The file self-balancing module is used to select a target storage node to perform a file quantity adjustment operation according to the difference in the number of file copies.

Images