File storage method, device, electronic device and computer readable medium

Abstract

The embodiments of the present disclosure disclose a file storage method, device, electronic device and computer-readable medium. A specific implementation of the method includes: in response to receiving a data file input by a user, dividing the data file according to a preset capacity threshold and the data capacity of the data file; for each data block in at least one data block and the first target slave node in the target slave node sequence, in response to the storage type of the data block being the first storage type, storing the data block in the first storage resource pool of the first target slave node; in response to the storage type of the data block being the second storage type, storing the data block in the second storage resource pool of the first target slave node. This implementation is related to big data, realizes the classified storage of data files, and improves the reading rate of small data files.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a file storage method, device, electronic device, and computer-readable medium. Background Art

[0002] With the rapid development of big data technology, data is showing an explosive growth trend. At present, the Hadoop system usually stores data files in mechanical hard disks.

[0003] However, the above storage method usually has the following technical problems: data files are not classified and stored. When the number of small data files (small data files may refer to files whose data capacity is less than the preset capacity) stored in the mechanical hard disk is large, the system's reading rate for small data files is reduced, which can easily cause system freezes when the access volume is large. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a file storage method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a file storage method, the method comprising: in response to receiving a data file input by a user, dividing the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block; for each data block in the at least one data block and the first target slave node in the target slave node sequence, performing the following storage steps: in response to the storage type of the data block being a first storage type, storing the data block in a first storage resource pool of the first target slave node; in response to the storage type of the data block being a second storage type, storing the data block in a second storage resource pool of the first target slave node.

[0007] Optionally, the method further includes: in response to detecting that the first storage resource pool of a slave node in the slave node set satisfies a resource merging condition, determining the first storage resource pool in the first storage resource pool corresponding to the slave node set that satisfies the resource merging condition as a first candidate storage resource pool, and obtaining a first candidate storage resource pool group; for each first candidate storage resource pool in the first candidate storage resource pool group, performing the following processing steps: sending file information of each data block stored in the first candidate storage resource pool to a preset transit server, wherein the file information includes a file name, a file type and an access frequency, and the transit server is used to read data blocks whose access frequency is less than or equal to a preset frequency, and classify and merge the read data blocks according to the file type of the data blocks.

[0008] Optionally, the above processing steps also include: in response to receiving the merged data file sent by the above transit server and the index information corresponding to the above merged data file, determining the slave node with the largest remaining storage space capacity corresponding to the slave node in the above slave node set as the first target slave node; storing the above merged data file in the second storage resource pool of the above first target slave node.

[0009] Optionally, the processing steps further include: in response to completion of storage of the merged data file, deleting each metadata information corresponding to the index information; in response to completion of storage of the merged data file, deleting each data block in the first storage resource pool in the slave node corresponding to the merged data file.

[0010] Optionally, the storage step further includes: in response to successful storage, backing up the data block to each target slave node in the target slave node sequence excluding the first target slave node.

[0011] Optionally, in response to receiving the data file input by the user, before dividing the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block, the method further includes: in response to receiving a write request for the data file input by the user, determining whether there is a user identifier corresponding to the above-mentioned user in a preset user identifier group; in response to determining that there is a user identifier corresponding to the above-mentioned user in the user identifier group, determining whether there is a file name corresponding to the above-mentioned data file in a preset directory tree; in response to determining that there is no file name corresponding to the above-mentioned data file in the above-mentioned user identifier group, receiving the data file input by the user.

[0012] Optionally, before executing the following storage steps for each data block in the at least one data block and the first target slave node in the target slave node sequence, the method further includes: marking the storage type of the data block in the at least one data block whose data capacity is less than the capacity threshold as a first storage type; marking the storage type of the data block in the at least one data block whose data capacity is equal to the capacity threshold as a second storage type.

[0013] Optionally, after marking the storage type of the data block in the at least one data block whose data capacity is equal to the capacity threshold as the second storage type, the method further includes: selecting a predetermined number of slave nodes in the slave node set in turn according to the slave node information set to obtain a target slave node sequence; and for each target slave node in the target slave node sequence excluding the first target slave node, establishing a transmission channel between the first target slave node and the target slave node.

[0014] Optionally, the slave node information in the above slave node information set includes the slave node name, the remaining storage space capacity of the slave node, the relative distance of the slave node and the heartbeat signal transmission time.

[0015] Optionally, the above-mentioned selecting a predetermined number of slave nodes in the slave node set in sequence according to the slave node information set includes: according to the remaining storage space capacity of the slave nodes included in the above-mentioned slave node information set, the slave node information in the above-mentioned slave node information set is sorted in descending order to obtain a slave node information sequence as a first slave node information sequence; in response to the presence of at least one first slave node information with the same remaining storage space capacity of the slave nodes included in the above-mentioned first slave node information sequence, according to the relative distances of the slave nodes included in the above-mentioned slave node information set, at least one first slave node information with the same remaining storage space capacity of the slave nodes in the above-mentioned first slave node information sequence is sorted in descending order to obtain a second slave node information sequence; in response to the presence of at least one second slave node information with the same relative distance of the slave nodes included in the above-mentioned second slave node information sequence, according to the heartbeat signal transmission time included in the above-mentioned slave node information set, at least one second slave node information with the same relative distance of the slave nodes in the above-mentioned second slave node information sequence is sorted in descending order to obtain a third slave node information sequence.

[0016] Optionally, the above-mentioned selecting a predetermined number of slave nodes in the slave node set in sequence according to the slave node information set also includes: selecting the above-mentioned predetermined number of slave node information in sequence from the above-mentioned third slave node information sequence to obtain a target slave node information sequence; and determining the slave node in the above-mentioned slave node set corresponding to the slave node name included in each target slave node information in the above-mentioned target slave node information sequence as the target slave node.

[0017] In a second aspect, some embodiments of the present disclosure provide a file storage device, comprising: a segmentation unit, configured to, in response to receiving a data file input by a user, segment the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block; a storage unit, configured to, for each data block in the at least one data block and a first target slave node in a target slave node sequence, perform the following storage steps: in response to the storage type of the data block being a first storage type, storing the data block in a first storage resource pool of the first target slave node; in response to the storage type of the data block being a second storage type, storing the data block in a second storage resource pool of the first target slave node.

[0018] Optionally, the device also includes: a storage resource pool determination unit, configured to, in response to detecting that a first storage resource pool of a slave node in a slave node set satisfies a resource merging condition, determine a first storage resource pool in the first storage resource pool corresponding to the above slave node set that satisfies the above resource merging condition as a first candidate storage resource pool, and obtain a first candidate storage resource pool group; a processing unit, configured to, for each first candidate storage resource pool in the above first candidate storage resource pool group, perform the following processing steps: send file information of each data block stored in the above first candidate storage resource pool to a preset transfer server, wherein the above file information includes a file name, a file type and an access frequency, and the above transfer server is used to Read data blocks with an access frequency less than or equal to a preset frequency, and classify and merge the read data blocks according to the file type of the data blocks; in response to receiving the merged data file sent by the above-mentioned transit server and the index information corresponding to the above-mentioned merged data file, determine the slave node with the largest remaining storage space capacity of the slave node corresponding to the above-mentioned slave node set as the first target slave node; store the above-mentioned merged data file in the second storage resource pool of the above-mentioned first target slave node; in response to the completion of the storage of the above-mentioned merged data file, delete the various metadata information corresponding to the above-mentioned index information; in response to the completion of the storage of the above-mentioned merged data file, delete the various data blocks in the first storage resource pool of the slave node corresponding to the above-mentioned merged data file.

[0019] Optionally, the storage unit further includes: a backup subunit configured to: in response to successful storage, back up the data block to each target slave node in the target slave node sequence excluding the first target slave node.

[0020] Optionally, before the splitting unit, the device also includes: a first determination unit, configured to determine whether there is a user identifier corresponding to the above-mentioned user in a preset user identifier group in response to receiving a write request for a data file input by a user; a second determination unit, configured to determine whether there is a file name corresponding to the above-mentioned data file in a preset directory tree in response to determining that there is a user identifier corresponding to the above-mentioned user in the above-mentioned user identifier group; and a receiving unit, configured to receive the data file input by the user in response to determining that there is no file name corresponding to the above-mentioned data file in the above-mentioned user identifier group.

[0021] Optionally, before splitting the order, the device also includes: a selection unit, configured to select a predetermined number of slave nodes in the slave node set in turn according to the slave node information set to obtain a target slave node sequence; an establishment unit, configured to establish a transmission channel between the above-mentioned first target slave node and the above-mentioned target slave node for each target slave node in the target slave node sequence excluding the above-mentioned first target slave node.

[0022] Optionally, the slave node information in the above slave node information set includes the slave node name, the remaining storage space capacity of the slave node, the relative distance of the slave node and the heartbeat signal transmission time.

[0023] Optionally, the selection unit is further configured to: perform descending order processing on the slave node information in the above-mentioned slave node information set according to the remaining storage space capacity of the slave nodes included in the above-mentioned slave node information set, and obtain a slave node information sequence as a first slave node information sequence; in response to the presence of at least one first slave node information with the same remaining storage space capacity of the slave nodes included in the above-mentioned first slave node information sequence, according to the relative distance of the slave nodes included in the above-mentioned slave node information set, perform descending order sorting on the at least one first slave node information with the same remaining storage space capacity of the slave nodes in the above-mentioned first slave node information sequence, and obtain a second slave node information sequence; in response to the presence of at least one second slave node information with the same relative distance of the slave nodes included in the above-mentioned second slave node information sequence, according to the heartbeat signal transmission time included in the above-mentioned slave node information set, perform descending order sorting on the at least one second slave node information with the same relative distance of the slave nodes in the above-mentioned second slave node information sequence, and obtain a third slave node information sequence.

[0024] Optionally, the selection unit is further configured to: select the above-mentioned predetermined number of slave node information in sequence from the above-mentioned third slave node information sequence to obtain a target slave node information sequence; and determine the slave node in the above-mentioned slave node set corresponding to the slave node name included in each target slave node information in the above-mentioned target slave node information sequence as the target slave node.

[0025] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0026] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0027] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the file storage method of some embodiments of the present disclosure, the classified storage of data files is realized, the reading rate of small data files is improved, and the jamming of the system is alleviated. Specifically, the reason for the jamming of the system is that the data files are not classified and stored. When the number of small data files stored in the mechanical hard disk is large, the reading rate of the small data files of the system is reduced, and when the access volume is large, it is easy to cause the jamming of the system. Based on this, the file storage method of some embodiments of the present disclosure, first, in response to receiving the data file input by the user, according to the preset capacity threshold and the data capacity of the above data file, the above data file is segmented to obtain at least one data block. Thereby, it is convenient to classify and store the data blocks after the segmentation in the subsequent step. Then, for each data block in the above at least one data block and the first target slave node in the target slave node sequence, in response to the storage type of the above data block being the first storage type, the above data block is stored in the first storage resource pool of the above first target slave node. Finally, in response to the storage type of the above data block being the second storage type, the above data block is stored in the second storage resource pool of the above first target slave node. As a result, classified storage of data blocks with different data capacities is achieved, alleviating the storage pressure of mechanical hard disks. Thus, classified storage of data files is achieved, the reading rate of small data files is improved, and the system jamming is alleviated. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0029] Figure 1 is a schematic diagram of an application scenario of the file storage method of some embodiments of the present disclosure;

[0030] Figure 2is a schematic diagram of another application scenario of the file storage method of some embodiments of the present disclosure

[0031] Figure 3 is a flow chart of some embodiments of the file storage method according to the present disclosure;

[0032] Figure 4 is a flowchart of other embodiments of the file storage method according to the present disclosure;

[0033] Figure 5 are flowcharts of some further embodiments of the file storage method according to the present disclosure;

[0034] Figure 6 is a schematic diagram of the structure of some embodiments of the file storage device according to the present disclosure;

[0035] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0037] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0038] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] The file storage method provided in the embodiment of the present disclosure can be applied to the business scenario of small file storage of HDFS in the Hadoop cluster. In addition, the file storage method provided in the embodiment of the present disclosure can also be applied to other existing or future business scenarios, which are not limited by the present disclosure.

[0042] In order to solve one or more of the problems described in the background technology section, when data files are stored in a mechanical hard disk, the following three methods are usually used to store the data files:

[0043] First, use Hadoop Archive to archive data files.

[0044] Second, use Sequence files with the small file name as the key and the file content as the value.

[0045] Third, parameter merging: During data collection and processing, set some system parameters provided by Hadoop to reduce the generation of small files.

[0046] However, although the first method reduces the number of small files on HDFS, it requires access to index data before reading files, and the reading efficiency of small files after merging is slower than before merging. The second method is only applicable to scenarios where a large number of small files are written at one time. The third method requires R&D personnel to be very familiar with the underlying source code of Hadoop, and each data processing task must set parameters separately to reduce the generation of small files, which cannot be applied on a large scale.

[0047] Therefore, in order to completely solve the drawbacks of the above three methods, and solve the technical problems raised by the background technology. The present invention provides a file storage method, which makes full use of the characteristics of two different storage media, SSD and HDD, to store small files and large files respectively. The SSD resource pool can be regarded as a high-speed storage area, and the HDD resource pool can be regarded as a low-speed storage area. Using SSD to store small files makes up for the shortcomings of HDFS frequently performing disk IO when reading small files. Using HDD to store large files converts random file reading into sequential reading as much as possible, thereby improving the reading performance of HDFS files. When the small files exceed a certain threshold, some small files with low access frequency are classified and merged with the help of a transit server, and small files of the same type are merged into a large file and stored in the HDD. The merged large file replaces the original small file, freeing up the high-speed storage space of the SSD.

[0048] In addition, the present invention realizes efficient use of storage performance of Hadoop cluster in two storage media, SSD and HDD. Small files with high frequency of access are stored on SSD, and small files with low frequency of access are merged and stored with large files on HDD. All small file merging operations are performed on the transit server, without any impact on the existing HDFS files, thus ensuring the high availability of the Hadoop cluster. There is no disk IO when reading small files on SSD, thus improving the access speed; large files (or large files after merging small files) are read sequentially on HDD, thus reducing the number of random reads. In addition, the memory load of Namenode is also reduced by triggering the merging of small files through thresholds. The problem of low storage access efficiency of the original HDFS file is solved, the pressure of Namenode managing files is effectively reduced, the storage utilization efficiency of small files is improved, and the overall performance of the Hadoop cluster is improved.

[0049] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0050] Figure 1 It is a schematic diagram of an application scenario of the file storage method according to some embodiments of the present disclosure.

[0051] exist Figure 1 In the application scenario, first, the computing device 101 can, in response to receiving the data file 102 input by the user, divide the data file according to the preset capacity threshold 103 and the data capacity of the data file 102 to obtain at least one data block 104. Then, the computing device 101 can perform the following storage steps for each data block 1041 in the at least one data block 104 and the first target slave node 1051 in the target slave node sequence 105: in response to the storage type of the data block 1041 being the first storage type, the data block 1041 is stored in the first storage resource pool of the first target slave node 1051; in response to the storage type of the data block 1041 being the second storage type, the data block 1041 is stored in the second storage resource pool of the first target slave node 1051.

[0052] It should be noted that the computing device 101 can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0053] It should be understood that Figure 1The number of computing devices in the embodiment is only illustrative. Any number of computing devices may be provided according to implementation requirements.

[0054] Figure 2 is a schematic diagram of another application scenario of the file storage method according to some embodiments of the present disclosure.

[0055] exist Figure 2 In the application scenario, the data flow process of the client client, slave node DataNode (target slave node), master node NameNode and transit server in the file storage method is described. Before introducing the interaction between the client client, slave node DataNode, master node NameNode and transit server, it should be noted that the client client, slave node DataNode and master node NameNode all belong to the Hadoop cluster. Here, the Hadoop cluster includes a distributed file system (HDFS). HDFS may include DataNode and NameNode. DataNode, as a slave node, mainly provides the function of data storage. The storage media used by the Datanode in the present invention are SSD (Solid State Drives) and HDD (Hard Disk Drive). The client client can be regarded as the entry point for writing or reading files in the Hadoop cluster. When writing files, Hadoop will split the files into block data blocks and store them on the Datanode as a logical unit. The transit server is a server built outside the Hadoop cluster to merge data blocks and process the merged metadata information. Among them, Figure 2 HDFSclient in the above code represents the client of HDFS. DistributedFileSystem can represent the API (Application Programming Interface) of the client. FSDataOutputStream can represent the API for the client to write data.

[0056] The data flow process between the client, the slave node DataNode (target slave node), the master node NameNode and the transit server is as follows:

[0057] 1. The client initiates a write request for the data file. Here, the client sends the write request for the data file to the NameNode through the DistributedFileSystem.

[0058] 2. After receiving the client's request, Namenode will first check the metadata directory tree (the preset target tree), check the permissions (determine whether there is a user ID corresponding to the above user in the preset user ID group) and determine whether the file to be written already exists (determine whether there is a file name corresponding to the above data file in the preset directory tree). If it already exists, the client's write request is rejected. If it does not exist, it responds to the client's request and allows writing (in response to determining that there is no file name corresponding to the above data file in the above user ID group, the data file input by the above user is received).

[0059] 3. After the client receives a response that the file can be written, it will cut the file to be written into blocks, and one or more data blocks will be generated after the cutting (in response to receiving the data file input by the user, the data file is cut into two blocks of 128MB+1MB according to the preset capacity threshold and the data capacity of the data file to obtain at least one data block). For example, a 129MB file will be cut into two blocks of 128MB+1MB, and a 64MB file will be cut into one block. In addition, if the data block after cutting is smaller than the default data block size of the Hadoop cluster (such as, the default data block size of the Hadoop cluster is 128MB), the file to be written (data block) is marked as a small file (the storage type of the data block in the at least one data block whose data capacity is smaller than the capacity threshold is marked as the first storage type, and in response to the storage type of the data block being the first storage type, the data block is stored in the first storage resource pool of the first target slave node). If it is consistent with the default data block size, the file to be written is marked as a large file (the storage type of the data block whose data capacity is equal to the capacity threshold in the at least one data block is marked as the second storage type, and in response to the storage type of the data block being the second storage type, the data block is stored in the second storage resource pool of the first target slave node). Then, the client sends a request to the Namenode again, asking the Namenode which block the data to be written should be stored in.

[0060] 4. After Namenode receives the request from the client to write a block, it will first detect the Natanode node information it saves, query the block address of the appropriate conditions and return it to the client (according to the slave node information set, a predetermined number of slave nodes are selected in turn from the slave node set to obtain the target slave node sequence). Here, the appropriate conditions are: First, the Datanode's disk is relatively idle. Second, the Datanode is close to the client, the network topology is relatively close, and the fewer switches, the better. Third, the Datanode is relatively active, and the transmission time of sending heartbeat signals to the Namenode is relatively short.

[0061] 5. After receiving the Datanode information, the client first establishes a network connection with a nearby Datanode, and then the Datanode establishes a transmission channel with the remaining slave nodes. After the channel is connected, it returns confirmation information to the client (for each target slave node in the target slave node sequence excluding the above-mentioned first target slave node, a transmission channel between the above-mentioned first target slave node and the above-mentioned target slave node is established), indicating that the channel is connected and data can be transmitted.

[0062] 6. After receiving the confirmation information, the client writes the data of the first block to the nearest Datanode through the network. After receiving the data, the nearest Datanode caches it first, and then determines whether the file to be written is a large file or a small file. If it is a small file, the data in the cache is written to the local SSD storage resource pool (in response to the storage type of the above data block being the first storage type, the above data block is stored in the first storage resource pool of the above first target slave node). If it is a large file, the cached data is written to the local HDD storage resource pool (in response to the storage type of the above data block being the second storage type, the above data block is stored in the second storage resource pool of the above first target slave node). Finally, the data in the cache is sent to the transmission channel, and the remaining Datanodes back up the data in the same way (in response to successful storage, the above data block is backed up to each target slave node in the target slave node sequence excluding the above first target slave node). After each Datanode writes a data block, it returns a confirmation message.

[0063] 7. After the first block is written, if the client still has remaining data blocks to be written, the client will start from step 3 and continue to execute the above steps 3-6 until the entire file is written. Finally, the output stream is closed and a completion signal is sent to the Namenode.

[0064] 8. When the number of small files in the SSD storage resource pool on the Datanode or the occupied storage space capacity exceeds a certain threshold, the Datanode immediately sends a small file merge request to the Namenode (in response to detecting that the first storage resource pool of the slave node in the slave node set meets the resource merging condition, the first storage resource pool in the first storage resource pool corresponding to the above slave node set that meets the above resource merging condition is determined as the first candidate storage resource pool, and the first candidate storage resource pool group is obtained).

[0065] 9. Namenode then sends the small file information (metadata information such as the storage location of small files) reported by Datanode to the transit server (sends the file information of each data block stored in the first alternative storage resource pool to the preset transit server. The file information includes file name, file type and access frequency. The transit server is used to read data blocks with an access frequency less than or equal to the preset frequency, and classify and merge the read data blocks according to the file type of the data blocks).

[0066] It should be noted that the transit server will identify the received small file information, identify the small files with low access frequency, and then read and load these small files with low access frequency from the datanode to the transit server (small files with high access frequency continue to remain in the SSD), and merge them into the preset classification file directory according to the type of small files loaded. At the same time, index information is established to extract the file name, data node location, data block location, offset offset and file length of each small file. Then merge the small files in each preset classification file directory.

[0067] 10. Namenode receives the merged file information and the index information of the original small files. Repeat steps 4-7 for the merged file information. At the same time, based on the index information of the original small files, find the location information, node information, file length, etc. of the replica files corresponding to these small files (in response to receiving the merged data file sent by the above-mentioned transit server and the index information corresponding to the above-mentioned merged data file, the slave node with the largest remaining storage space capacity corresponding to the above-mentioned slave node set is determined as the first target slave node; the above-mentioned merged data file is stored in the second storage resource pool of the above-mentioned first target slave node).

[0068] 11. After the merged file is written, Namenode deletes all metadata information (file name, file length, data block location, node location, number of copies, etc.) of the original small file according to the index information of the original small file. And notify Datanode to synchronously delete the original small file in the SSD storage resource pool to completely release the storage space (in response to the completion of the storage of the above merged data file, delete the metadata information corresponding to the above index information; in response to the completion of the storage of the above merged data file, delete the data blocks in the first storage resource pool in the slave node corresponding to the above merged data file).

[0069] It should be noted that the above steps 1-11 are only a general description of the relevant data flow in the file storage method of the present disclosure. For specific data flow details, please refer to Figure 3-Figure 5 Description of the embodiments in .

[0070] Continue to refer Figure 3 , shows a process 300 of some embodiments of the file storage method according to the present disclosure. The file storage method comprises the following steps:

[0071] Step 301 , in response to receiving a data file input by a user, the data file is segmented according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block.

[0072] In some embodiments, the execution subject of the file storage method (for example Figure 1 The computing device 101 shown in the figure can, in response to receiving a data file input by a user, divide the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block. Here, the preset capacity threshold can be a capacity threshold for dividing the data file. Here, the execution subject of the file storage method can refer to a Hadoop cluster. Here, the data file can also include a file name. In practice, the execution subject can divide the data file according to a preset capacity threshold. For example, the data capacity of data file "A" can be 149MB. The capacity threshold can be 128MB. The data file can be divided into a data block with a data capacity of 128MB and a data block with a data capacity of 21MB.

[0073] Step 302: For each data block in the at least one data block and the first target slave node in the target slave node sequence, the following storage steps are performed:

[0074] Step 3021: In response to the storage type of the data block being the first storage type, the data block is stored in the first storage resource pool of the first target slave node.

[0075] In some embodiments, the execution subject may store the data block in the first storage resource pool of the first target slave node in response to the storage type of the data block being the first storage type. Here, the first storage type may refer to the type of data block whose data capacity is equal to the capacity threshold. Here, the target slave node in the target slave node sequence may refer to the slave node whose heartbeat signal transmission time from the slave node to the master node of the execution subject is less than or equal to the preset time. Here, the slave node may refer to the DataNode of the distributed file system (HDFS) in the Hadoop cluster. Here, the master node may refer to the NameNode of the distributed file system (HDFS) in the Hadoop cluster. Here, the heartbeat signal transmission time may refer to the time for the slave node to transmit the data block to the master node. Here, the first storage resource pool may refer to a storage disk for storing data blocks of the first storage type. For example, the first storage resource pool may be an SSD (Solid State Drives) of a slave node. For example, the execution subject may store the data block "A" in the first storage resource pool SSD of the first target slave node "001" in response to the storage type of the data block "A" being the first storage type.

[0076] Step 3022: In response to the storage type of the data block being the second storage type, the data block is stored in the second storage resource pool of the first target slave node.

[0077] In some embodiments, the above-mentioned execution subject may store the above-mentioned data block in the second storage resource pool of the above-mentioned first target slave node in response to the storage type of the above-mentioned data block being the second storage type. Here, the second storage type may refer to the type of data block whose data capacity is less than the capacity threshold. Here, the second storage resource pool may refer to a storage disk for storing data blocks of the second storage type. For example, the second storage resource pool may be the HDD (Hard Disk Drive) of the slave node. For example, the above-mentioned execution subject may store the above-mentioned data block "A" in the second storage resource pool HDD of the above-mentioned first target slave node "001" in response to the storage type of the above-mentioned data block "A" being the second storage type.

[0078] Optionally, in response to successful storage, the data block is backed up to each target slave node in the target slave node sequence excluding the first target slave node.

[0079] In some embodiments, in response to the storage success, the execution subject may back up the data block to the disk of each target slave node in the target slave node sequence except the first target slave node. For example, the target slave node sequence may be "target slave node 001; target slave node 002; target slave node 003". The execution subject may back up the data block "A" to the disk of the target slave node 002 and the target slave node 003.

[0080] In this way, the backup of data blocks is achieved. When the access volume to the data blocks is large, the data blocks can be read in a dispersed manner from each target slave node, so as to ease the reading pressure of the system when the access volume is large.

[0081] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the file storage method of some embodiments of the present disclosure, the classified storage of data files is realized, the reading rate of small data files is improved, and the jamming of the system is alleviated. Specifically, the reason for the jamming of the system is that the data files are not classified and stored. When the number of small data files stored in the mechanical hard disk is large, the reading rate of the small data files of the system is reduced, and when the access volume is large, it is easy to cause the jamming of the system. Based on this, the file storage method of some embodiments of the present disclosure, first, in response to receiving the data file input by the user, according to the preset capacity threshold and the data capacity of the above data file, the above data file is segmented to obtain at least one data block. Thereby, it is convenient to classify and store the data blocks after the segmentation in the subsequent step. Then, for each data block in the above at least one data block and the first target slave node in the target slave node sequence, in response to the storage type of the above data block being the first storage type, the above data block is stored in the first storage resource pool of the above first target slave node. Then, in response to the storage type of the above data block being the second storage type, the above data block is stored in the second storage resource pool of the above first target slave node. As a result, classified storage of data blocks with different data capacities is achieved, alleviating the storage pressure of mechanical hard disks. Thus, classified storage of data files is achieved, the reading rate of small data files is improved, and the system jamming is alleviated.

[0082] Further references Figure 4 , showing some other embodiments of the file storage method according to the present disclosure. The file storage method comprises the following steps:

[0083] Step 401 : in response to receiving a data file write request input by a user, determining whether there is a user identification corresponding to the user in a preset user identification group.

[0084] In some embodiments, the execution subject of the file storage method (for example Figure 1The computing device 101 shown in the figure can determine whether there is a user identification corresponding to the above-mentioned user in a preset user identification group in response to receiving a write request for a data file input by a user. Here, the user identification in the preset user identification group can be a user identification that is pre-set and stored in a local server. Here, the write request can include a user identification. In practice, the above-mentioned execution subject can determine whether there is a user identification in the preset user identification group that is the same as the user identification included in the above-mentioned write request.

[0085] Step 402: In response to determining that a user identification corresponding to the user exists in the user identification group, determining whether a file name corresponding to the data file exists in a preset directory tree.

[0086] In some embodiments, the execution subject may determine whether there is a file name corresponding to the data file in a preset directory tree in response to determining that there is a user identification corresponding to the user in the user identification group. Here, the preset directory tree may be a directory tree constructed according to the file names of each data file stored locally. In practice, the execution subject may determine whether there is a file name identical to the file name of the data file in a preset directory tree in response to determining that there is a user identification corresponding to the user in the user identification group.

[0087] Step 403: In response to determining that the file name corresponding to the data file does not exist in the user identification group, receiving the data file input by the user.

[0088] In some embodiments, the execution subject may receive the data file input by the user in response to determining that the file name corresponding to the data file does not exist in the user identification group. Here, the data file may include but is not limited to: file name, file type and data capacity.

[0089] Step 404 , in response to receiving the data file input by the user, the data file is segmented according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block.

[0090] In some embodiments, the specific implementation of step 404 and the technical effects brought about can be referred to Figure 3 The corresponding step 301 in the embodiments will not be described in detail here.

[0091] Step 405: Mark the storage type of the data block whose data capacity is less than the capacity threshold in the at least one data block as the first storage type.

[0092] In some embodiments, the execution subject may mark the storage type of the data block whose data capacity is less than the capacity threshold in the at least one data block as the first storage type. For example, the data capacity of data block "A" may be: 21MB, which is less than the capacity threshold "128MB", and the storage type of data block "A" is marked as the first storage type.

[0093] Step 406: Mark the storage type of the data block in the at least one data block whose data capacity is equal to the capacity threshold as the second storage type.

[0094] In some embodiments, the execution subject may mark the storage type of the data block whose data capacity is equal to the capacity threshold in the at least one data block as the second storage type. For example, the data capacity of data block "A" may be: 128MB, which is equal to the capacity threshold "128MB", and the storage type of data block "A" is marked as the second storage type.

[0095] Step 407 : According to the slave node information set, a predetermined number of slave nodes are selected in sequence from the slave node set to obtain a target slave node sequence.

[0096] In some embodiments, the above-mentioned execution subject may select a predetermined number of slave nodes in the slave node set in sequence according to the slave node information set to obtain a target slave node sequence. Here, the slave node information in the slave node information set may include: slave node name, slave node remaining storage space capacity, slave node relative distance and heartbeat signal transmission time. Here, the slave node remaining storage space capacity may refer to the storage capacity of the remaining storage space of the slave node. Here, the slave node relative distance may refer to the number of switches spanned from the slave node to the master node of the above-mentioned execution subject. Here, the heartbeat signal transmission time may refer to the time for the slave node to transmit a data block to the master node.

[0097] In practice, according to the slave node information set, the above execution subject may sequentially select a predetermined number of slave nodes from the slave node set through the following steps:

[0098] The first step is to process the slave node information in the above slave node information set in descending order according to the remaining storage space capacity of the slave nodes included in the above slave node information set, and obtain a slave node information sequence as the first slave node information sequence. In practice, the above execution subject can process the slave node information in the above slave node information set in descending order from large to small according to the value of the remaining storage space capacity of the slave nodes included in the above slave node information set, and obtain a slave node information sequence as the first slave node information sequence.

[0099] For example, the slave node information set could be:

[0100] {[Slave node name: slave node 001; slave node remaining storage space capacity: 550MB; slave node relative distance: 1; heartbeat signal transmission time: 0.1 seconds];

[0101] [Slave node name: slave node 002; slave node remaining storage space capacity: 530MB; slave node relative distance: 2; heartbeat signal transmission time: 0.2 seconds];

[0102] [Slave node name: slave node 003; slave node remaining storage space capacity: 530MB; slave node relative distance: 1; heartbeat signal transmission time: 0.2 seconds];

[0103] [Slave node name: slave node 004; slave node remaining storage space capacity: 500MB; slave node relative distance: 2; heartbeat signal transmission time: 0.3 seconds];

[0104] [Slave node name: slave node 005; slave node remaining storage space capacity: 500MB; slave node relative distance: 2; heartbeat signal transmission time: 0.2 seconds]}.

[0105] The execution subject may process the slave node information in the slave node information set in descending order according to the values ​​of the remaining storage space capacity of the slave nodes included in the slave node information set, and obtain the slave node information sequence {[slave node name: slave node 001; remaining storage space capacity of slave node: 550MB; relative distance of slave node: 1; heartbeat signal transmission time: 0.1 seconds]; [slave node name: slave node 002; remaining storage space capacity of slave node: 530MB; relative distance of slave node: 2; heartbeat signal transmission time: 0.2 seconds]; [Slave node name: slave node 003; Remaining storage space capacity of slave node: 530MB; Relative distance of slave node: 1; Heartbeat signal transmission time: 0.2 seconds]; [Slave node name: slave node 004; Remaining storage space capacity of slave node: 500MB; Relative distance of slave node: 2; Heartbeat signal transmission time: 0.3 seconds]; [Slave node name: slave node 005; Remaining storage space capacity of slave node: 500MB; Relative distance of slave node: 2; Heartbeat signal transmission time: 0.2 seconds]} as the first slave node information sequence.

[0106] In the second step, in response to the existence of at least one first slave node information with the same remaining storage space capacity of the slave nodes in the above-mentioned first slave node information sequence, according to the relative distance of the slave nodes included in the above-mentioned slave node information set, at least one first slave node information with the same remaining storage space capacity of the slave nodes in the above-mentioned first slave node information sequence is sorted in descending order to obtain a second slave node information sequence.

[0107] In practice, in response to the presence of at least one first slave node information with the same remaining storage space capacity of slave nodes in the above-mentioned first slave node information sequence, the above-mentioned execution entity can sort the at least one first slave node information with the same remaining storage space capacity of slave nodes in the above-mentioned first slave node information sequence in descending order from large to small according to the numerical values ​​of the relative distances of the slave nodes included in the above-mentioned slave node information set to obtain a second slave node information sequence.

[0108] For example, the first slave node information sequence may be:

[0109] {[Slave node name: slave node 001; slave node remaining storage space capacity: 550MB; slave node relative distance: 1; heartbeat signal transmission time: 0.1 seconds];

[0110] [Slave node name: slave node 002; slave node remaining storage space capacity: 530MB; slave node relative distance: 2; heartbeat signal transmission time: 0.2 seconds];

[0111] [Slave node name: slave node 003; slave node remaining storage space capacity: 530MB; slave node relative distance: 1; heartbeat signal transmission time: 0.2 seconds];

[0112] [Slave node name: slave node 004; slave node remaining storage space capacity: 500MB; slave node relative distance: 2; heartbeat signal transmission time: 0.3 seconds];

[0113] [Slave node name: slave node 005; slave node remaining storage space capacity: 500MB; slave node relative distance: 2; heartbeat signal transmission time: 0.2 seconds]}.

[0114] The execution subject may sort at least one first slave node information with the same remaining storage space capacity of the slave node in the first slave node information sequence in descending order according to the values ​​of the relative distances of the slave nodes included in the slave node information set, to obtain a second slave node information sequence:

[0115] {[Slave node name: slave node 001; slave node remaining storage space capacity: 550MB; slave node relative distance: 1; heartbeat signal transmission time: 0.1 seconds];

[0116] [Slave node name: slave node 003; slave node remaining storage space capacity: 530MB; slave node relative distance: 1; heartbeat signal transmission time: 0.2 seconds];

[0117] [Slave node name: slave node 002; slave node remaining storage space capacity: 530MB; slave node relative distance: 2; heartbeat signal transmission time: 0.2 seconds];

[0118] [Slave node name: slave node 004; slave node remaining storage space capacity: 500MB; slave node relative distance: 2; heartbeat signal transmission time: 0.3 seconds];

[0119] [Slave node name: slave node 005; slave node remaining storage space capacity: 500MB; slave node relative distance: 2; heartbeat signal transmission time: 0.2 seconds]}.

[0120] In the third step, in response to the presence of at least one second slave node information with the same relative distance from the slave nodes in the above-mentioned second slave node information sequence, according to the heartbeat signal transmission time included in the above-mentioned slave node information set, the at least one second slave node information with the same relative distance from the slave nodes in the above-mentioned second slave node information sequence is sorted in descending order to obtain a third slave node information sequence.

[0121] In practice, in response to the presence of at least one second slave node information with the same relative distance from the slave nodes in the above-mentioned second slave node information sequence, the above-mentioned execution entity can sort the at least one second slave node information with the same relative distance from the slave nodes in the above-mentioned second slave node information sequence in descending order from small to large according to the numerical value of the heartbeat signal transmission time included in the above-mentioned slave node information set to obtain a third slave node information sequence.

[0122] The fourth step is to sequentially select the predetermined number of slave node information from the third slave node information sequence to obtain a target slave node information sequence. Here, there is no restriction on the setting of the predetermined number.

[0123] In the fifth step, a slave node in the above slave node set corresponding to a slave node name included in each target slave node information in the above target slave node information sequence is determined as a target slave node.

[0124] Step 408: For each target slave node in the target slave node sequence excluding the first target slave node, a transmission channel between the first target slave node and the target slave node is established.

[0125] In some embodiments, for each target slave node in the target slave node sequence excluding the first target slave node, the execution entity may establish a transmission channel between the first target slave node and the target slave node by wired connection or wireless connection.

[0126] Step 409, for each data block in the at least one data block and the first target slave node in the target slave node sequence, perform the following storage steps: in response to the storage type of the data block being the first storage type, store the data block in the first storage resource pool of the first target slave node; in response to the storage type of the data block being the second storage type, store the data block in the second storage resource pool of the first target slave node.

[0127] In some embodiments, the specific implementation of step 409 and the technical effects brought about can be referred to Figure 3 The corresponding step 302 in the embodiments will not be described in detail here.

[0128] from Figure 4 It can be seen that Figure 3 Compared with the description of some corresponding embodiments, Figure 4 In some corresponding embodiments, the process 400 firstly, in response to receiving a write request for a data file input by a user, determines whether there is a user identification corresponding to the above user in a preset user identification group. Thus, it is determined whether the data file input by the user has the permission to store. Secondly, in response to determining that there is a user identification corresponding to the above user in the above user identification group, determines whether there is a file name corresponding to the above data file in a preset directory tree. Then, in response to determining that there is no file name corresponding to the above data file in the above user identification group, receives the data file input by the user. Thus, repeated storage of data files with the same file name is avoided, resulting in a waste of storage resources. Then, in response to receiving the data file input by the user, the data file is segmented according to a preset capacity threshold and the data capacity of the above data file to obtain at least one data block. Then, the storage type of the data block whose data capacity is less than the above capacity threshold in the above at least one data block is marked as a first storage type; the storage type of the data block whose data capacity is equal to the above capacity threshold in the above at least one data block is marked as a second storage type. Thus, it is convenient to classify and store the segmented data blocks in the subsequent process. Then, according to the slave node information set, a predetermined number of slave nodes are selected in sequence from the slave node set to obtain a target slave node sequence. Thus, it is convenient to quickly store the segmented data blocks. Finally, for each target slave node in the target slave node sequence excluding the first target slave node, a transmission channel between the first target slave node and the target slave node is established. Thus, by establishing the transmission channel, the efficiency of data backup between slave nodes is improved.

[0129] Further references Figure 5 , showing some further embodiments of the file storage method according to the present disclosure. The file storage method comprises the following steps:

[0130] Step 501 , in response to receiving a data file input by a user, the data file is segmented according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block.

[0131] Step 502, for each data block in the at least one data block and the first target slave node in the target slave node sequence, perform the following storage steps: in response to the storage type of the data block being the first storage type, store the data block in the first storage resource pool of the first target slave node; in response to the storage type of the data block being the second storage type, store the data block in the second storage resource pool of the first target slave node.

[0132] In some embodiments, the specific implementation of steps 501-502 and the technical effects brought about can be referred to Figure 3 The corresponding steps 301-302 in the embodiments are not described in detail here.

[0133] Step 503, in response to detecting that the first storage resource pool of the slave node in the slave node set meets the resource merging condition, the first storage resource pool in the first storage resource pool corresponding to the above slave node set that meets the above resource merging condition is determined as the first candidate storage resource pool to obtain a first candidate storage resource pool group.

[0134] In some embodiments, the execution subject of the file storage method (for example Figure 1 The computing device 101 shown in the figure can, in response to detecting that the first storage resource pool of the slave node in the slave node set meets the resource merging condition, determine the first storage resource pool in the first storage resource pool corresponding to the above slave node set that meets the resource merging condition as the first candidate storage resource pool, and obtain a first candidate storage resource pool group. Here, the resource merging condition may refer to "the number of data blocks stored in the first storage resource pool is greater than or equal to the preset number, or the remaining storage space capacity of the first storage resource pool is less than or equal to the preset storage capacity". In practice, the above execution subject can, in response to detecting that the first storage resource pool of the slave node in the slave node set meets the resource merging condition "the number of data blocks stored in the first storage resource pool is greater than or equal to the preset number, or the remaining storage space capacity of the first storage resource pool is less than or equal to the preset storage capacity", determine the first storage resource pool in the first storage resource pool corresponding to the above slave node set that meets the resource merging condition "the number of data blocks stored in the first storage resource pool is greater than or equal to the preset number, or the remaining storage space capacity of the first storage resource pool is less than or equal to the preset storage capacity" as the first candidate storage resource pool, and obtain a first candidate storage resource pool group. Here, there is no restriction on the setting of the preset number. Here, there is no restriction on the setting of the preset storage capacity.

[0135] Step 504: for each first candidate storage resource pool in the first candidate storage resource pool group, perform the following processing steps:

[0136] Step 5041: Send the file information of each data block stored in the first candidate storage resource pool to a preset transfer server.

[0137] In some embodiments, the above-mentioned execution entity may send the file information of each data block stored in the above-mentioned first alternative storage resource pool to a preset transit server. The above-mentioned file information includes file name, file type and access frequency. The above-mentioned transit server is used to read data blocks whose access frequency is less than or equal to the preset frequency, and classify and merge the read data blocks according to the file type of the data block. Here, the file information may also include but is not limited to at least one of the following: file length, data block position, slave node position, offset. Here, the data block position may refer to the storage position of the data block in the first storage resource pool. Here, the slave node position may refer to the position of the slave node in the above-mentioned execution entity. Here, the access frequency may refer to the number of times the data block is read.

[0138] In practice, first, after receiving the file information of each data block sent by the above-mentioned execution subject, the transit server selects the file information including the access frequency less than or equal to the preset frequency from the file information of each data block as the candidate file information, and obtains the candidate file information group. Then, the transit server can read the candidate data block group from the first candidate storage resource pool according to the data block position included in the candidate file information group. Here, the candidate data blocks in the candidate data block group correspond to the candidate file information of the above-mentioned candidate file information group. Then, the transit server can place the candidate data blocks in the read candidate data block group into the corresponding preset classification file directory according to the file type. For example, if the file type is a picture class, the candidate data blocks of the picture class are placed in the file directory of the preset picture class. At the same time, the file name, slave node position, data block position, offset and file length of the data blocks under each preset classification file directory are extracted to combine into index information. Then, the transit server can merge the data blocks under each preset classification file directory to generate a merged data file. Finally, the transit server may send the generated merged data file and index information to the above-mentioned execution entity in sequence.

[0139] Step 5042, in response to receiving the merged data file and the index information corresponding to the merged data file sent by the above-mentioned transit server, determine the slave node with the largest remaining storage space capacity of the corresponding slave node in the above-mentioned slave node set as the first target slave node.

[0140] In some embodiments, the execution entity may determine the slave node with the largest remaining storage space capacity corresponding to the slave node in the slave node set as the first target slave node in response to receiving the merged data file and index information corresponding to the merged data file sent by the transit server.

[0141] Step 5043: store the merged data file in the second storage resource pool of the first target slave node.

[0142] In some embodiments, the execution entity may store the merged data file in the second storage resource pool of the first target slave node.

[0143] Step 5044: in response to the completion of storing the merged data file, each metadata information corresponding to the index information is deleted.

[0144] In some embodiments, the execution subject may delete the metadata information pointed to by the index information stored locally in response to the completion of the storage of the merged data file. Here, the metadata information may include but is not limited to at least one of the following: file name, data node location, data block location, offset, and access frequency.

[0145] Step 5045: in response to the completion of storage of the merged data file, each data block in the first storage resource pool in the slave node corresponding to the merged data file is deleted.

[0146] In some embodiments, the execution subject may delete each data block in the first storage resource pool in the slave node corresponding to the merged data file in response to the completion of storage of the merged data file. In practice, the execution subject may delete each data block in the first storage resource pool in the slave node corresponding to the merged data file according to the file name, slave node position and data block position included in the index information.

[0147] from Figure 5 It can be seen that Figure 3 Compared with the description of some corresponding embodiments, Figure 5 In some corresponding embodiments, the process 500 can classify and merge some data blocks with low access frequency with the help of the transit server after the data blocks in the first storage resource pool exceed a certain threshold, and can merge the data blocks of the same file type into a merged data file (large file) and store it in the HDD. At the same time, delete the data blocks corresponding to the above merged data file in the SSD of the first storage resource pool to release the high-speed storage space of the first storage resource pool. Thereby, the pressure of managing files is effectively reduced, the storage efficiency of small data files is improved, and the overall performance of the Hadoop cluster is improved.

[0148] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a file storage device. These device embodiments are similar to Figure 3 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0149] like Figure 6 As shown, the file storage device 600 of some embodiments includes: a splitting unit 601 and a storage unit 602. The splitting unit 601 is configured to, in response to receiving a data file input by a user, split the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block; the storage unit 602 is configured to perform the following storage steps for each data block in the at least one data block and the first target slave node in the target slave node sequence: in response to the storage type of the data block being the first storage type, storing the data block in the first storage resource pool of the first target slave node; in response to the storage type of the data block being the second storage type, storing the data block in the second storage resource pool of the first target slave node.

[0150] Optionally, the device 600 also includes: a storage resource pool determination unit, configured to, in response to detecting that a first storage resource pool of a slave node in the slave node set satisfies a resource merging condition, determine a first storage resource pool in the first storage resource pool corresponding to the above slave node set that satisfies the above resource merging condition as a first candidate storage resource pool, and obtain a first candidate storage resource pool group; a processing unit, configured to, for each first candidate storage resource pool in the above first candidate storage resource pool group, perform the following processing steps: send file information of each data block stored in the above first candidate storage resource pool to a preset transit server, wherein the above file information includes a file name, a file type and an access frequency, and the above transit server uses The method comprises the following steps: for reading data blocks with a read access frequency less than or equal to a preset frequency, and classifying and merging the read data blocks according to the file types of the data blocks; in response to receiving the merged data file sent by the above-mentioned transit server and the index information corresponding to the above-mentioned merged data file, determining the slave node with the largest remaining storage space capacity of the slave node corresponding to the above-mentioned slave node set as the first target slave node; storing the above-mentioned merged data file in the second storage resource pool of the above-mentioned first target slave node; in response to the completion of storing the above-mentioned merged data file, deleting each metadata information corresponding to the above-mentioned index information; in response to the completion of storing the above-mentioned merged data file, deleting each data block in the first storage resource pool of the slave node corresponding to the above-mentioned merged data file.

[0151] Optionally, the storage unit 602 further includes: a backup subunit configured to: in response to successful storage, back up the data block to each target slave node in the target slave node sequence excluding the first target slave node.

[0152] Optionally, before the splitting unit 601, the device 600 also includes: a first determination unit, configured to determine whether there is a user identifier corresponding to the above-mentioned user in a preset user identifier group in response to receiving a write request for a data file input by a user; a second determination unit, configured to determine whether there is a file name corresponding to the above-mentioned data file in a preset directory tree in response to determining that there is a user identifier corresponding to the above-mentioned user in the above-mentioned user identifier group; and a receiving unit, configured to receive the data file input by the user in response to determining that there is no file name corresponding to the above-mentioned data file in the above-mentioned user identifier group.

[0153] Optionally, before the splitting unit 601, the device 600 also includes: a selection unit, configured to select a predetermined number of slave nodes in the slave node set in sequence according to the slave node information set to obtain a target slave node sequence; an establishment unit, configured to establish a transmission channel between the above-mentioned first target slave node and the above-mentioned target slave node for each target slave node in the target slave node sequence excluding the above-mentioned first target slave node.

[0154] Optionally, the slave node information in the above slave node information set includes the slave node name, the remaining storage space capacity of the slave node, the relative distance of the slave node and the heartbeat signal transmission time.

[0155] Optionally, the selection unit is further configured to: perform descending order processing on the slave node information in the above-mentioned slave node information set according to the remaining storage space capacity of the slave nodes included in the above-mentioned slave node information set, and obtain a slave node information sequence as a first slave node information sequence; in response to the presence of at least one first slave node information with the same remaining storage space capacity of the slave nodes included in the above-mentioned first slave node information sequence, according to the relative distance of the slave nodes included in the above-mentioned slave node information set, perform descending order sorting on the at least one first slave node information with the same remaining storage space capacity of the slave nodes in the above-mentioned first slave node information sequence, and obtain a second slave node information sequence; in response to the presence of at least one second slave node information with the same relative distance of the slave nodes included in the above-mentioned second slave node information sequence, according to the heartbeat signal transmission time included in the above-mentioned slave node information set, perform descending order sorting on the at least one second slave node information with the same relative distance of the slave nodes in the above-mentioned second slave node information sequence, and obtain a third slave node information sequence.

[0156] Optionally, the selection unit is further configured to: select the above-mentioned predetermined number of slave node information in sequence from the above-mentioned third slave node information sequence to obtain a target slave node information sequence; and determine the slave node in the above-mentioned slave node set corresponding to the slave node name included in each target slave node information in the above-mentioned target slave node information sequence as the target slave node.

[0157] It is understood that the units described in the device 600 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the units included therein, and will not be described in detail here.

[0158] Reference below Figure 7 , which shows an electronic device (eg, Figure 1 The electronic devices in some embodiments of the present disclosure may include but are not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0159] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0160] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0161] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0162] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0163] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0164] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to receiving a data file input by a user, the data file is segmented according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block; for each data block in the at least one data block and the first target slave node in the target slave node sequence, the following storage steps are performed: in response to the storage type of the data block being the first storage type, the data block is stored in the first storage resource pool of the first target slave node; in response to the storage type of the data block being the second storage type, the data block is stored in the second storage resource pool of the first target slave node.

[0165] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a segmentation unit and a storage unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the segmentation unit may also be described as "a unit that divides the received data file according to a preset capacity threshold to obtain at least one data block".

[0168] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0169] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A file storage method, comprising: In response to receiving a data file input by a user, the data file is segmented according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block; For each data block in the at least one data block and the first target slave node in the sequence of target slave nodes, performing the following storing steps: In response to the storage type of the data block being the first storage type, storing the data block in the first storage resource pool of the first target slave node, wherein the target slave node in the target slave node sequence refers to a slave node whose transmission time of the heartbeat signal from the slave node to the master node is less than or equal to a preset time; In response to the storage type of the data block being the second storage type, storing the data block in a second storage resource pool of the first target slave node; In response to successful storage, the data block is backed up to each target slave node in the target slave node sequence excluding the first target slave node.

2. The method according to claim 1, wherein: The method further comprises: In response to detecting that a first storage resource pool of a slave node in the slave node set meets a resource merging condition, determining a first storage resource pool in the first storage resource pool corresponding to the slave node set that meets the resource merging condition as a first candidate storage resource pool, to obtain a first candidate storage resource pool group; For each first candidate storage resource pool in the first candidate storage resource pool group, the following processing steps are performed: The file information of each data block stored in the first alternative storage resource pool is sent to a preset relay server, wherein the file information includes a file name, a file type and an access frequency. The relay server is used to read data blocks whose access frequency is less than or equal to a preset frequency, and to classify and merge the read data blocks according to the file type of the data blocks.

3. The method according to claim 2, wherein: The processing steps also include: In response to receiving the merged data file and the index information corresponding to the merged data file sent by the transit server, determining the slave node with the largest remaining storage space capacity of the corresponding slave node in the slave node set as the first target slave node; The merged data file is stored in a second storage resource pool of the first target slave node.

4. The method according to claim 3, wherein: The processing steps also include: In response to completion of storing the merged data file, deleting each metadata information corresponding to the index information; In response to the completion of storing the merged data file, each data block in the first storage resource pool in the slave node corresponding to the merged data file is deleted.

5. The method according to claim 1, wherein: Before the step of, in response to receiving the data file input by the user, dividing the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block, the method further comprises: In response to receiving a data file write request input by a user, determining whether there is a user identifier corresponding to the user in a preset user identifier group; In response to determining that a user identification corresponding to the user exists in the user identification group, determining whether a file name corresponding to the data file exists in a preset directory tree; In response to determining that the file name corresponding to the data file does not exist in the user identification group, receiving the data file input by the user.

6. The method according to claim 1, wherein: Before performing the following storing step for each data block in the at least one data block and the first target slave node in the target slave node sequence, the method further comprises: Marking the storage type of a data block whose data capacity is less than the capacity threshold in the at least one data block as a first storage type; The storage type of the data block whose data capacity is equal to the capacity threshold in the at least one data block is marked as the second storage type.

7. The method according to claim 6, wherein: After marking the storage type of the data block whose data capacity is equal to the capacity threshold in the at least one data block as the second storage type, the method further includes: According to the slave node information set, a predetermined number of slave nodes are selected in sequence from the slave node set to obtain a target slave node sequence; For each target slave node in the target slave node sequence excluding the first target slave node, a transmission channel between the first target slave node and the target slave node is established.

8. The method according to claim 7, wherein: The slave node information in the slave node information set includes the slave node name, the remaining storage space capacity of the slave node, the relative distance from the slave node and the heartbeat signal transmission time; as well as The step of sequentially selecting a predetermined number of slave nodes from the slave node set according to the slave node information set includes: According to the remaining storage space capacity of the slave nodes included in the slave node information set, the slave node information in the slave node information set is processed in descending order to obtain a slave node information sequence as a first slave node information sequence; In response to the existence of at least one first slave node information with the same remaining storage space capacity of slave nodes in the first slave node information sequence, at least one first slave node information with the same remaining storage space capacity of slave nodes in the first slave node information sequence is sorted in descending order according to the relative distances of the slave nodes included in the slave node information set, to obtain a second slave node information sequence; In response to the presence of at least one second slave node information with the same relative distance from the slave nodes in the second slave node information sequence, at least one second slave node information with the same relative distance from the slave nodes in the second slave node information sequence is sorted in descending order according to the heartbeat signal transmission time included in the slave node information set to obtain a third slave node information sequence.

9. The method according to claim 8, wherein: The step of sequentially selecting a predetermined number of slave nodes from the slave node set according to the slave node information set further comprises: Selecting the predetermined number of slave node information in sequence from the third slave node information sequence to obtain a target slave node information sequence; A slave node in the slave node set corresponding to a slave node name included in each target slave node information in the target slave node information sequence is determined as a target slave node.

10. A file storage device, applied to the method according to claim 1, comprising: a splitting unit configured to, in response to receiving a data file input by a user, split the data file according to a preset capacity threshold and the data capacity of the data file to obtain at least one data block; The storage unit is configured to perform the following storage steps for each data block in the at least one data block and the first target slave node in the target slave node sequence: in response to the storage type of the data block being the first storage type, storing the data block in the first storage resource pool of the first target slave node; in response to the storage type of the data block being the second storage type, storing the data block in the second storage resource pool of the first target slave node.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images