Distributed storage methods, devices, equipment, and storage media for files

Abstract

This disclosure provides a distributed file storage method, apparatus, device, and storage medium, applicable to the field of data storage technology. The method includes collecting storage information from multiple nodes; constructing a mapping relationship between the storage information and corresponding nodes; and writing and / or reading files based on the storage information and the mapping relationship. In this way, files that cannot be stored on a single machine partition can be automatically split and merged, enabling the storage of extremely large files and significantly improving the utilization rate of each disk.

Description

Technical Field

[0001] This disclosure relates to the field of data storage technology, and in particular to a distributed storage method, apparatus, device, and storage medium for files. Background Technology

[0002] In a big data environment, the volume of metadata is also enormous, and the performance of metadata access is crucial to the overall performance of the file system. Common metadata management can be divided into centralized and distributed metadata management architectures.

[0003] Centralized metadata management architectures use a single metadata server, which is simple to implement but suffers from single points of failure. Distributed metadata management architectures, on the other hand, distribute metadata across multiple nodes, thus solving the performance bottleneck of the metadata server and improving the scalability of the metadata management architecture. However, they are more complex to implement and introduce metadata consistency issues.

[0004] Another approach is a distributed architecture without a metadata server, which organizes data through online algorithms, eliminating the need for a dedicated metadata server. However, this architecture faces significant challenges in ensuring data consistency and is relatively complex to implement. File directory traversal operations are inefficient, and it lacks global file system monitoring and management capabilities.

[0005] Therefore, there is an urgent need to build an efficient and reasonable distributed storage method for files. Summary of the Invention

[0006] This disclosure provides a method, apparatus, device, and storage medium for distributed file storage.

[0007] According to a first aspect of this disclosure, a distributed storage method for files is provided. The method includes:

[0008] Collect storage information from multiple nodes;

[0009] Construct a mapping relationship between the stored information and the corresponding nodes;

[0010] Based on the storage information and the mapping relationship, files are written to and / or read from.

[0011] In addition to the aspects described above and any possible implementations, a further implementation is provided, wherein the storage information includes the current remaining storage space size and inode information of the corresponding node;

[0012] Writing files based on the storage information and the mapping relationship includes:

[0013] Receive the file to be stored;

[0014] Compare the file size of the file to be stored with the maximum current remaining storage space.

[0015] If the file size is less than or equal to the maximum current remaining storage space, then the file to be stored is written to the node corresponding to the maximum remaining storage space.

[0016] If the file size is greater than the maximum remaining storage space, then the splitting strategy of the file to be stored is determined based on the current remaining storage space size and the inode information;

[0017] The file to be stored is split according to the splitting strategy and written to the corresponding node.

[0018] In addition to the aspects and any possible implementations described above, an implementation is further provided in which the inode information includes the total number of inodes and the current number of inodes;

[0019] The step of determining the splitting strategy for the file to be stored based on the current remaining storage space size and the inode information includes:

[0020] Based on the current remaining storage space and the file size, multiple first storage nodes are determined;

[0021] Calculate the ratio between the current number of inodes and the total number of inodes in the first storage node;

[0022] If the ratio is greater than or equal to a preset threshold, the corresponding first storage node is taken as the first node of the file to be stored, and the remaining storage space of the first node is taken as the first segmentation size of the file to be stored.

[0023] The second partition size of the file to be stored is determined based on the remaining storage space of the nodes other than the first node in the first storage node and the size of the remaining file to be stored.

[0024] Use the first segmentation size and the second segmentation size as the segmentation strategy;

[0025] The size of the remaining file to be stored is the difference between the file size and the first segment size.

[0026] In addition to the aspects described above and any possible implementations, a further implementation is provided, wherein determining a plurality of first storage nodes based on the current remaining storage space size and the file size includes:

[0027] The current remaining storage space sizes are randomly combined and summed.

[0028] The node corresponding to any combination whose sum is greater than or equal to the file size is taken as the first storage node.

[0029] In addition to the aspects described above and any possible implementations, a further implementation is provided, wherein determining the second partition size of the file to be stored based on the remaining storage space size of the nodes other than the first node in the first storage node and the size of the remaining file to be stored includes:

[0030] The nodes in the first storage node other than the first node are identified as the second storage node;

[0031] Random values ​​are selected from each of the remaining storage spaces within the range that is less than or equal to the current size of the second storage node, and then summed.

[0032] Use any set of random values ​​whose sum equals the size of the remaining files to be stored as the second split size.

[0033] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the step of splitting the file to be stored according to the splitting strategy and writing it to the corresponding nodes includes:

[0034] The file to be stored is divided into multiple segmented files based on the first segmentation size and the second segmentation size;

[0035] Write the segmented file to the corresponding node; and update the mapping relationship.

[0036] In addition to the aspects and any possible implementations described above, a further implementation is provided, which reads a file based on the storage information and the mapping relationship, including:

[0037] Receive a file read command; the file read command includes the file to be read;

[0038] Based on the file to be read, determine the corresponding mapping relationship, and further determine the corresponding node and inode information;

[0039] The file is read from the corresponding node based on the determined inode information.

[0040] According to a second aspect of this disclosure, a distributed storage device for files is provided. The device includes:

[0041] The information acquisition module is used to collect stored information from multiple nodes;

[0042] A mapping relationship construction module is used to construct the mapping relationship between the stored information and the corresponding nodes;

[0043] The file storage module is used to write and / or read files according to the storage information and the mapping relationship.

[0044] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0045] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in the first aspect of this disclosure.

[0046] The distributed storage method, apparatus, device, and storage medium for files provided in the embodiments of this disclosure write and / or read files by constructing a mapping relationship between the storage information of each node and the corresponding node. Files that cannot be stored in a single machine partition are automatically split and merged to achieve ultra-large file storage and greatly improve the utilization rate of each disk.

[0047] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0048] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0049] Figure 1 A flowchart illustrating a distributed storage method for files according to an embodiment of the present disclosure is shown;

[0050] Figure 2 A flowchart illustrating the writing of a file according to an embodiment of the present disclosure is shown;

[0051] Figure 3 A flowchart illustrating a file splitting strategy according to an embodiment of the present disclosure is shown.

[0052] Figure 4 A block diagram of a distributed storage device for files according to embodiments of the present disclosure is shown;

[0053] Figure 5 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0055] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0056] In this disclosure, the master server constructs a mapping relationship between the storage information of each node and the corresponding node to dynamically merge multiple file systems on a single machine into a huge storage space, thereby achieving ultra-large file storage and greatly improving the utilization of each disk.

[0057] Figure 1 A flowchart of a distributed storage method 100 for files according to an embodiment of the present disclosure is shown. Method 100 includes:

[0058] Step 110: Collect storage information from multiple nodes.

[0059] In some embodiments, the nodes are distributed nodes, and the storage information includes the current remaining storage space size and inode information of the corresponding node. The inode information includes the total number of inodes and the current number of inodes. Specifically, disk partition and inode information on each node can be collected via remote commands. The disk partition information on each node includes the current remaining storage space size. An inode, also known as an index node, is used to store files on a hard drive. The smallest unit of a hard drive is a sector, each sector being 512 bytes in size. To improve hard drive read efficiency, the system reads multiple sectors consecutively, and these sectors are combined into a block. A block is the smallest unit of file access, with a block size of 4KB. File attributes, creation time, permissions, block size, and number of blocks constitute inode information.

[0060] In some embodiments, the information in an inode includes: file type, permissions, UID (User ID of the file owner, i.e., user identifier or user identification code), GID (Group ID of the file, i.e., unique identifier of the file), number of links, file size, distinct timestamps, pointers to data blocks pointing to the file on disk, and other data related to the file.

[0061] Step 120: Construct the mapping relationship between the stored information and the corresponding nodes.

[0062] In some embodiments, a mapping relationship between the storage information obtained in step 110 and the corresponding node is constructed and saved in the memory of the main server in the form of a hashmap data structure.

[0063] Step 130: Write and / or read the file according to the storage information and the mapping relationship.

[0064] In some embodiments, such as Figure 2 The flowchart shown illustrates the process of writing a file, including the following steps:

[0065] Step 210: Receive the file to be stored.

[0066] In some embodiments, the file to be stored is received, and the file size of the file to be stored is extracted from it.

[0067] Step 220: Compare the file size of the file to be stored with the maximum current remaining storage space.

[0068] In some embodiments, it is necessary to compare the maximum value between the file size of the file to be stored obtained in step 210 and the current remaining storage space to determine whether there is a node that can be directly stored.

[0069] Step 230: If the file size is less than or equal to the maximum current remaining storage space size, then write the file to be stored into the node corresponding to the maximum remaining storage space size.

[0070] In some embodiments, if the file size of the file to be stored is less than or equal to the maximum value of the current remaining storage space, it means that there is a node that can directly store the file, and the file to be stored is directly stored in any node that meets this condition.

[0071] Step 240: If the file size is greater than the maximum remaining storage space size, then determine the splitting strategy of the file to be stored based on the current remaining storage space size and the inode information.

[0072] In some embodiments, if the file size of the file to be stored is greater than the maximum value of the current remaining storage space, it means that there is no node that can directly store the file, the file is too large, and the file to be stored needs to be split.

[0073] In some embodiments, such as Figure 3 The flowchart of the file splitting strategy shown below includes the following steps:

[0074] Step 310: Determine multiple first storage nodes based on the current remaining storage space and file size.

[0075] In some embodiments, step 310 specifically includes: randomly combining the current remaining storage space sizes and summing them; selecting any combination whose sum is greater than or equal to the file size as the first storage node. For example, if the file size to be stored is 100GB, the current remaining storage space sizes are randomly combined, meaning that the sum of the current remaining storage space sizes is greater than or equal to 100GB. Examples of combinations that satisfy this condition include: Combination 1 - Node a (current remaining storage space size is 50GB) + Node b (current remaining storage space size is 30GB) + Node c (current remaining storage space size is 20GB); Combination 2 - Node d (current remaining storage space size is 70GB) + Node e (current remaining storage space size is 40GB), etc. Then, a group is randomly selected as the first storage node; for example, Combination 1 is selected.

[0076] Step 320: Calculate the ratio between the current number of inodes of the first storage node and the total number of inodes.

[0077] In some embodiments, assuming combination one is selected as the first storage node, then it is necessary to calculate the ratio x between the current number of inodes of each node a, node b and node c and the total number of inodes of that node.

[0078] Step 330: If the ratio is greater than or equal to a preset threshold, then the corresponding first storage node is taken as the first node of the file to be stored, and the remaining storage space of the first node is taken as the first segmentation size of the file to be stored.

[0079] In some embodiments, if one of the ratios x calculated for each node in step 320 is greater than or equal to a preset threshold (e.g., 95%), meaning that the current inode count of that node is nearing saturation, then that node is preferentially selected as the first node. For example, for node b, the remaining storage space of node b, 30GB, is used as the first partition size. This allows for consideration of both the current remaining storage space and whether the current inode count is nearing saturation, and determines priorities to achieve efficient use of disk space and improve utilization. It avoids situations where, for example, only one inode remains, writing a file much smaller than the current remaining storage space of that node would inevitably lead to a waste of storage resources for that node.

[0080] In some embodiments, if none of the ratios x calculated in step 320 for each node are greater than or equal to a preset threshold, then the current remaining storage space size corresponding to each node in this combination, along with the file size, is used to determine the partition size. For example, in combination one above, 50GB, 30GB, and 20GB are directly used as the partition size for the file to be stored. Of course, if the current remaining storage space size of the selected first storage node is greater than the file size of the file to be stored, then any value can be partitioned, as long as the sum of the partitioned sizes is equal to the file size of the file to be stored and does not exceed the current remaining storage space size of its own node. For example, in combination two above, possible random partitioning strategies could be: 65GB+35GB, 70GB+30GB, 68GB+32GB, etc.

[0081] Step 340: Determine the second partition size of the file to be stored based on the remaining storage space of the nodes other than the first node in the first storage node and the size of the remaining file to be stored.

[0082] In some embodiments, step 340 specifically includes: determining the nodes other than the first node in the first storage node as the second storage node; randomly selecting values ​​within the range of the current remaining storage space size of the second storage node and summing them; and taking any set of random values ​​whose sum is equal to the size of the remaining files to be stored as the second split size.

[0083] In some embodiments, such as combination one described above, if the remaining storage space of node b (30GB) is selected as the first partition size, then after partitioning, the remaining file size to be stored is 70GB. Random values ​​are then taken from the remaining remaining storage space of nodes a and c, and summed. Any random value whose sum equals the remaining file size to be stored is used as the second partition size. Here, due to the limitations of the example above, the remaining remaining storage space of nodes a and c is just sufficient to store the file. Of course, in actual practice, various situations may arise; it is sufficient to follow the rules set above, without being limited by the example.

[0084] Step 350: Use the first segmentation size and the second segmentation size as the segmentation strategy.

[0085] The size of the remaining file to be stored is the difference between the file size and the first segment size.

[0086] In some embodiments, the first and second partition sizes determined according to the above steps are used as the partitioning strategy for the file to be stored.

[0087] This allows for the integrated use of idle disk partitions, improving the utilization and flexibility of physical disks and effectively saving on hardware costs.

[0088] Step 250: The file to be stored is split according to the splitting strategy and written to the corresponding nodes.

[0089] In some embodiments, the files to be stored are split according to the determined splitting strategy and written to the corresponding nodes for storage.

[0090] In some embodiments, to facilitate the merging of the split files read subsequently, the split files can be numbered according to the splitting order, and the files can be merged according to the numbering.

[0091] Therefore, by monitoring the nodes of the entire storage cluster and monitoring the resource utilization of all nodes and partitions in real time, we can use this as a basis for file distribution. This not only ensures that the files to be stored can be completely written, but also improves the disk space utilization of each node.

[0092] Based on the above embodiments, in another embodiment provided in this disclosure, reading the file includes: receiving a file reading command; the file reading command includes a file to be read; determining a corresponding mapping relationship based on the file to be read, and further determining the corresponding node and inode information; and reading the file from the corresponding node based on the determined inode information.

[0093] In some embodiments, a file read command is received, and the corresponding mapping relationship is determined based on the filename of the file to be read in the file read command. Alternatively, an ID can be generated for the file based on its filename during storage, serving as a unique identifier for finding the mapping relationship. Then, the node containing the file to be read and its inode information are found from the mapping relationship. Finally, based on the inode information, the block storage block containing the file data is located, and the data is read. In this way, a file can be automatically copied to different nodes, improving the high availability of file replicas across nodes.

[0094] Based on the above implementation methods, in another implementation method provided in this disclosure, the method further includes: if a storage node crashes and may cause the loss of the files stored on that node, the node sends the stored files to the master server, and the master server arranges for the files to be stored on other nodes. The storage rules are as described in the above embodiments and will not be repeated here.

[0095] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0096] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0097] Figure 4 A block diagram of a distributed storage device 400 for files according to an embodiment of the present disclosure is shown. Figure 4 As shown, the device 400 includes:

[0098] The information acquisition module 410 is used to collect stored information from multiple nodes;

[0099] The mapping relationship construction module 420 is used to construct the mapping relationship between the stored information and the corresponding nodes;

[0100] The file storage module 430 is used to write and / or read files according to the storage information and the mapping relationship.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0102] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0103] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0104] Electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in ROM 502 or a computer program loaded into RAM 503 from storage unit 508. RAM 503 can also store various programs and data required for the operation of electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. I / O interface 505 is also connected to bus 504.

[0105] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0106] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the distributed storage method for files. For example, in some embodiments, the distributed storage method for files can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the distributed storage method for files described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the distributed storage method for files by any other suitable means (e.g., by means of firmware).

[0107] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0111] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0112] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0113] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0114] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A distributed file storage method, characterized in that, include: Collect storage information from multiple nodes; Construct a mapping relationship between the stored information and the corresponding nodes; Based on the storage information and the mapping relationship, files are written and / or read; The storage information includes the current remaining storage space size and inode information of the corresponding node; Writing files based on the storage information and the mapping relationship includes: Receive the file to be stored; Compare the file size of the file to be stored with the maximum current remaining storage space. If the file size is less than or equal to the maximum current remaining storage space, then the file to be stored is written to the node corresponding to the maximum remaining storage space. If the file size is greater than the maximum remaining storage space, then the splitting strategy of the file to be stored is determined based on the current remaining storage space size and the inode information; The file to be stored is split according to the splitting strategy and written to the corresponding nodes; The inode information includes the total number of inodes and the current number of inodes; The step of determining the splitting strategy for the file to be stored based on the current remaining storage space size and the inode information includes: Based on the current remaining storage space and the file size, multiple first storage nodes are determined; Calculate the ratio between the current number of inodes and the total number of inodes in the first storage node; If the ratio is greater than or equal to a preset threshold, the corresponding first storage node is taken as the first node of the file to be stored, and the remaining storage space of the first node is taken as the first segmentation size of the file to be stored. The second partition size of the file to be stored is determined based on the remaining storage space of the nodes other than the first node in the first storage node and the size of the remaining file to be stored. Use the first segmentation size and the second segmentation size as the segmentation strategy; The size of the remaining file to be stored is the difference between the file size and the first segment size.

2. The method according to claim 1, characterized in that, The step of determining multiple first storage nodes based on the current remaining storage space size and the file size includes: The current remaining storage space sizes are randomly combined and summed. The node corresponding to any combination whose sum is greater than or equal to the file size is taken as the first storage node.

3. The method according to claim 1, characterized in that, The step of determining the second partition size of the file to be stored based on the remaining storage space of the nodes other than the first node in the first storage node and the size of the remaining file to be stored includes: The nodes in the first storage node other than the first node are identified as the second storage node; Random values ​​are selected from each of the remaining storage spaces within the range that is less than or equal to the current size of the second storage node, and then summed. Use any set of random values ​​whose sum equals the size of the remaining files to be stored as the second split size.

4. The method according to claim 1, characterized in that, The step of splitting the file to be stored according to the splitting strategy and writing it to the corresponding node includes: The file to be stored is divided into multiple segmented files based on the first segmentation size and the second segmentation size; Write the segmented file to the corresponding node; and update the mapping relationship.

5. The method according to claim 1, characterized in that, Based on the stored information and the mapping relationship, file reading is performed, including: Receive a file read command; the file read command includes the file to be read; Based on the file to be read, determine the corresponding mapping relationship, and further determine the corresponding node and inode information; The file is read from the corresponding node based on the determined inode information.

6. A distributed storage device for files, characterized in that, The distributed storage device is used to perform the method of any one of claims 1-5, comprising: The information acquisition module is used to collect stored information from multiple nodes; A mapping relationship construction module is used to construct the mapping relationship between the stored information and the corresponding nodes; The file storage module is used to write and / or read files according to the storage information and the mapping relationship.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

Images