[0041] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.
[0042] The present invention optimizes the above-mentioned problems existing in HDFS, enhances the stability of the distributed file system, and greatly improves the performance of the system. In addition, the entire system has better scalability and is more conducive to distributed File system deployment and application.
[0043] figure 1 It is a structural diagram of the distributed file system of the present invention.
[0044] Such as figure 1 As shown, the system includes a core control node and multiple sub-clusters, and each sub-cluster includes a sub-cluster control node and multiple data nodes, where:
[0045] The core control node is used to receive a file access request containing a file name from the client, parse the file name to determine the sub-cluster to which the file belongs, and issue the file access request to the sub-cluster control node in the sub-cluster ;
[0046] The sub-cluster control node is used to schedule the storage resource allocation of the data nodes in the sub-cluster to which it belongs, and establish an index to record the correspondence between the file name of the file stored in the data node of the sub-cluster and the data node where the file is located, and The data node related to the file access request is determined based on the file access request issued by the core control node and the corresponding relationship, and the determined data node identifier is notified to the client through forwarding of the core control node.
[0047] Data nodes are used to store files and access files according to the schedule of the client or the sub-cluster control node of the sub-cluster.
[0048] The data node is also used to receive the direct file access request determined according to the data node identifier cached by the client, and provide the client with direct file access based on the direct file access request.
[0049] In one embodiment:
[0050] The file access request includes: a file read request, a file delete request, or a file write request, etc.
[0051] In one embodiment:
[0052] The core control node is used to convert individual characters in the file name into corresponding numbers one by one according to the American Standard Code for Information Interchange (ASCII) code table, sum all the corresponding numbers, and divide the sum result by the total number of sub-clusters to get The remainder result of is the identifier of the sub-cluster to which the file belongs.
[0053] Specifically, the core control node can be composed of a computer or server with massive data processing capabilities, used for task scheduling of the entire distributed file system, responsible for judging the sub-cluster where the requested file is located, and controlling the sub-cluster connected to it according to the judgment result The nodes exchange information and complete the information forwarding from the client information to the control node in the specific sub-cluster.
[0054] The sub-cluster control node in each sub-cluster can be composed of a computer or server with massive data processing capabilities. It is responsible for scheduling the resource allocation of all data nodes managed by it, and establishing an index to record the name of the stored file and the data node where it is located. Correspondence.
[0055] The data nodes in each sub-cluster can be composed of computers or servers with data processing capabilities to store specific data files in the system. They store and retrieve data according to the schedule of the client or the control node of the cluster, and regularly Send a list of the blocks they store to the sub-cluster control nodes.
[0056] Further, the corresponding relationship between the determined data node identifier and the file name included in the file access request may be cached on the client; when the client subsequently has a file access request for the file corresponding to the file name, it directly connects to the corresponding The data node identified by the data node to access the file.
[0057] Further, after directly accessing the data node corresponding to the data node identification to access the file, determine whether the file data is valid according to the verification code, if it is valid, confirm that the access is valid, and if it is invalid, send to the sub-cluster of the data node The control node sends a file data request, the sub-cluster control node controls the backup data node to access the backup file data, and the backup file data provided by the backup data node is synchronized to the data node.
[0058] It can be seen that the present invention mainly adopts a small cluster lightweight system solution.
[0059] The starting point of the present invention is to control the continuous increase of data nodes, and the adoption of the sub-cluster method will not cause the memory shortage of the control nodes and the single-point bottleneck problem of I/O. That is, the existing HDFS structure is regarded as a sub-cluster of the present invention, and each sub-cluster has a unique sub-cluster control node and all data nodes associated with it. In this way, there will be many sub-clusters in the entire system, and for coordinated control between different sub-clusters, the present invention introduces a higher-level core control node.
[0060] When the client accesses the distributed file system, the core control node is responsible for task scheduling. Through the analysis and processing of the accessed file name, it determines the sub-cluster where the file is located, and then sends the task to the control in the sub-cluster Node. When the data node in the sub-cluster completes the task, it will return the result to the core control node through the sub-cluster control node, and then the core control node will give feedback to the client.
[0061] The core control node can determine the sub-cluster where the file is located through a variety of analysis and processing methods. For example, when the client needs to write a new file in the distributed file storage system, it requests the core control node to allocate the written sub-cluster. The core control node first converts the string of the file name into a number, and then divides it according to the The remainder method selects a specific sub-cluster, and then requests the sub-cluster control node of the specific sub-cluster to create a new data block.
[0062] For example, if the full name of the file to be written is /home/admin/demo.c, then the individual characters are converted into corresponding numbers one by one according to the ASCII code table, namely: 47(/)+104(h)+111( o)+109(m)+101(e)+47(/)+97(a)+100(d)+109(m)+105(i)+110(n)+47(/)+100( d)+101(e)+109(m)+111(o)+46(.)+99(c)=1653
[0063] If there are a total of 20 sub-clusters in the cluster, the result of taking the remainder is 1653/20=13, that is, the file storage location is the 13th sub-cluster.
[0064] After determining the sub-cluster where the file is stored, the data node where the data is stored in the sub-cluster can be determined based on various existing methods, which is not repeated in the present invention.
[0065] After saving data based on the above method, you can take out the data based on the similar method described above.
[0066] Through the above analysis, it can be seen that the existing distributed file system is mainly composed of Google's GFS and Hadoop's HDFS, both of which adopt similar technologies in system architecture, and the design ideas are basically the same. Both the Master end adopts a single node method, so that the system performance is largely determined by the Master node. In most scenarios, the design of a single control node can still meet the needs of the system, but in large-scale applications With the continuous growth of data volume and the expansion of storage scale, the system continues to increase data block nodes, and the amount of metadata increases accordingly, especially in various application scenarios such as pictures, videos, documents, and voices. Nodes need to manage a large number of small file metadata, making it overwhelmed and becoming the bottleneck of the entire system.
[0067] In the design of the present invention, HDFS is used as the basis for improvement, the original single Master node (NameNode node) function is split, part of the function is moved up to the new system control node, and part of the function is delivered to the subordinate data node .
[0068] For those applications that require low-latency access, the original HDFS is not suitable, because HDFS is designed for large throughput data, which comes at the cost of a certain delay. HDFS has a single Master design, and all file requests go through the single Master. When there are many requests, a larger delay will inevitably occur. After the HDFS is improved in the present invention, the pressure on each functional node is reduced, and the file access speed is accelerated, thereby effectively reducing the time delay.
[0069] In addition, due to the limited performance of the control node, the scale of the entire distributed file system is limited. After the improvement in the present invention, the pressure on the control nodes (core control nodes and sub-cluster control nodes) at each level is reduced, so that the entire system can be Expand in the form of sub-clusters. Moreover, the expansion method is also very simple, and has little effect on the original system part, and only needs to configure the new subsystem information in the core control node.
[0070] Based on the above detailed analysis, the present invention also proposes a data access method of the distributed file system. The system includes a core control node and multiple sub-clusters. Each sub-cluster includes a sub-cluster control node and multiple data nodes. The sub-cluster control node schedules the storage resource allocation of the data nodes in the sub-cluster to which it belongs, and establishes an index to record the The correspondence between the file name of the file stored in the data node of the sub-cluster and the data node where the file is located; the data node stores the file;
[0071] The method includes:
[0072] The core control node receives the file access request containing the file name from the client, parses the file name to determine the sub-cluster to which the file belongs, and issues the file access request to the sub-cluster control node in the sub-cluster;
[0073] The sub-cluster control node determines the data node related to the file access request based on the file access request issued by the core control node and the corresponding relationship, and informs the client of the determined data node identifier through the core control node's forwarding;
[0074] The data node accesses the file according to the schedule of the client or the sub-cluster control node of the sub-cluster to which it belongs.
[0075] In one embodiment:
[0076] The parsing the file name to determine the sub-cluster to which the file belongs includes:
[0077] According to the American Standard Code for Information Interchange (ASCII) code table, the individual characters in the file name are converted into corresponding numbers one by one, all corresponding numbers are summed, and the sum result is divided by the total number of sub-clusters, and the remaining result is the file The ID of the sub-cluster to which it belongs.
[0078] In one embodiment, the method further includes:
[0079] Cache the correspondence between the determined data node identifier and the file name included in the file access request on the client;
[0080] When the client subsequently has a file access request for the file corresponding to the file name, it directly connects to the data node corresponding to the data node identifier to access the file.
[0081] In one embodiment, the method further includes:
[0082] After directly accessing the data node corresponding to the data node identification to access the file, determine whether the file data is valid according to the verification code, if it is valid, confirm that the access is valid, if it is invalid, send it to the sub-cluster control node of the data node The file data request is controlled by the sub-cluster control node to access the backup file data from the backup data node, and the backup file data provided by the backup data node is synchronized to the data node.
[0083] figure 2 It is a flow chart of data reading of the distributed file system according to the present invention.
[0084] When the client reads a file, it first sends a file read request containing the file name to the core control node. The core control node converts the string of the file name into a corresponding number, and then locates the file where the file is located according to the remainder method. Cluster and return the sub-cluster identifier to the client, and the client can request the file from the sub-cluster control node corresponding to the sub-cluster identifier.
[0085] Then, the client establishes a connection with the sub-cluster control node server, and the sub-cluster control node server finds the corresponding data node in the disk, and returns metadata information to the client.
[0086] Then, the client requests data from the corresponding data node server, the file is divided into fragments of the same size, and the process returns all the fragments of the file concurrently for the client. Like the original HDFS operation, when the end of a block is read, the link to the data node will be broken, and then the next data node will be selected to obtain the next data block.
[0087] Or, when the client reads a file, it first sends a file read request containing the file name to the core control node. The core control node converts the character string of the file name into a corresponding number, and then locates the file according to the remainder method The control node of the sub-cluster determines the specific data node, and then returns the data node identification to the client. Then, the client directly accesses the data node corresponding to the data node identifier to access specific files.
[0088] If the client already knows the sub-cluster identifier (in the link or in the client cache), it does not need to go through the core control node, and can directly send a file read request to the control node server in the designated cluster to reduce one I/O operation.
[0089] When the client connects directly to the data node and finds the corresponding block to read the file, it will first verify the check code to check whether the read data is valid, if it is valid, it will read directly, if it is invalid, it will need to The sub-cluster control node sends a request to read the data from the backup data node server, and then the backup data node synchronizes the valid data to the main data node.
[0090] image 3 It is a flowchart of data writing in the distributed file system according to the present invention.
[0091] Such as image 3 As shown, when the client needs to write a file in the distributed file storage system, it requests the core control node to allocate a small cluster for writing. The core control node first converts the string of the file name into a number, and then divides it according to The remainder method selects a specific sub-cluster and requests the sub-cluster control node to create a new data block;
[0092] Then, the sub-cluster control node completes a series of verification work, and after confirming that the client is allowed to write the file, it sends a data block creation instruction to the data node, and returns the block address of the data node to the client.
[0093] Next, the client establishes a connection with the data node and requests to write the file to the data block allocated by the sub-cluster control node. The data node allocates the physical address that can be written in the block, sets the offset ID, and divides the file into the same size. Fragmentation and concurrent upload to the upload buffer of the data node.
[0094] When the upload is completed, the client can consider that the writing is complete, and can disconnect from the data node, and subsequent operations will be transparent to the client.
[0095] When the data node receives the file cache from the client, when it is relatively idle, the files are sequentially and asynchronously written to the data block of the hot backup data node, and the data node is connected to the hot backup data node, and the backup data is written in the same way Node. After the writing of the primary and secondary data nodes is completed, the primary data node notifies the sub-cluster control node that the writing is successful and updates the metadata, and the data node deletes the upload cache space occupied by the file.
[0096] The data node contacts the index server to write the file index into the index system. If one of the primary data node or the backup data node fails to write from the upload cache space to the disk, the primary data node will request the sub-cluster control node to allocate another data block for writing.
[0097] Figure 4 It is a data deletion flowchart of the distributed file system according to the present invention.
[0098] Such as Figure 4 As shown, when the client requests to delete a file in the distributed file storage system, it will send a request to the core control node. The core control node converts the file name into a number and uses the remainder method to determine the sub-cluster where the file is located. Cluster control node connection. The subsequent operations are the same as the steps for deleting files in the original HDFS, so I won't repeat them here.
[0099] In summary, the system proposed by the present invention includes a core control node and multiple sub-clusters, and each sub-cluster includes a sub-cluster control node and multiple data nodes. The core control node is used to receive a file containing a file name from a client. Access request, parse the file name to determine the sub-cluster to which the file belongs, and send the file access request to the sub-cluster control node in the sub-cluster; the sub-cluster control node is used to schedule the sub-cluster to which it belongs The storage resource allocation of the data nodes is indexed to record the correspondence between the file name of the file stored in the data node of the sub-cluster and the data node where the file is located, and based on the file access request issued by the core control node and the said The corresponding relationship determines the data node related to the file access request, and informs the client of the determined data node identifier through the forwarding of the core control node; the data node is used to store the file and is based on the client or the sub-cluster to which it belongs. The cluster controls the scheduling of nodes to access files. It can be seen that the present invention optimizes the above-mentioned problems in HDFS, enhances the stability of the distributed file system, and greatly improves the system performance. In addition, the entire system has better scalability and more Conducive to the deployment and application of distributed file systems.
[0100] In the technical scheme of the present invention, HDFS is used as the basis for improvement, the original single Master node (NameNode) function is split, some functions are moved up to the new system control node, and some functions are delivered to the subordinate data node. For those applications that require low-latency access, the original HDFS is not suitable, because HDFS is designed for large throughput data, which comes at the cost of a certain delay. With the design of HDFS single master, all requests for files must go through it. When there are many requests, a larger delay will inevitably occur. After the HDFS is improved in the present invention, the pressure on each functional node is reduced, and the file access speed is accelerated, thereby effectively reducing the time delay.
[0101] In addition, due to the limited performance of the control node, the scale of the entire distributed file system is limited. After the improvement in the present invention, the pressure on the control nodes (core control nodes and sub-cluster control nodes) at each level is reduced, so that the entire system can be Expand in the form of sub-clusters. Moreover, the expansion method is also very simple, and has little effect on the original system part, and only needs to configure the new subsystem information in the core control node.
[0102] The foregoing descriptions are only preferred embodiments of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc., made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
