Method for data processing in distributed file system and the distributed file system
A distributed file and file technology, applied in the field of data processing, can solve problems such as long time, and achieve the effect of shortening the recovery time of downtime
Active Publication Date: 2018-07-27
TENCENT TECH (SHENZHEN) CO LTD +1
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0008] However, since a single master node corresponds to multiple data nodes, when a downtime occurs, the master node needs to obtain information from each data node. This process is a one-to-many information collection process, which takes a long time
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
example
[0054] see Figure 5 , is an example of a flow chart of data uploading based on a distributed file system in the present invention, which includes the following steps:
[0055] Step 501, the master node receives a data upload request including a file path sent by a client.
[0056] Step 502, the master node determines the corresponding file ID for the file path, determines the meta server ID corresponding to the file ID, and feeds back the determined file ID and meta server ID to the client.
[0057] Data upload can be divided into new method or append method. The new method is to create a new file ID for the file path, and the append method is to add data under the original file ID corresponding to the file path. It is also possible to carry the upload method in the data upload request, specifically:
[0058] If the upload mode is to cover, then the master node determines the corresponding file ID for the file path, including: the master node creates a new file ID for the f...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a method for carrying out data processing by a distributed file system and the distributed file system. The system comprises node servers and a plurality of meta-information servers; each meta-information server is used for sending a data block information acquisition request comprising a meta server ID (Identity) to each node server, receiving a file ID, a node address and data block attribute information from each node server, storing a second mapping relation between each file ID and the corresponding node address in a memory, and storing the data block attribute information in the corresponding node addresses in the memory after downtime restart; each node server is used for receiving the data block information acquisition request from each meta-information server, determining whether a corresponding file is stored according to the corresponding meta server ID and feeding back the file ID, the node address and the data block attribute information of the corresponding file to the corresponding meta-information server if the corresponding file is stored. According to the scheme adopted by the invention, recovery time after downtime restart can be shortened.
Description
technical field [0001] The invention relates to data processing technology, in particular to a data processing method of a distributed file system and the distributed file system. Background technique [0002] see figure 1 , is a schematic structural diagram of a distributed file system (DFS, Distributed File System) in the prior art, and the system includes a client, multiple data nodes, and a single master node. [0003] The master node stores the first mapping relationship between the file path and the file identifier (ID, IDentity) on the local hard disk, and stores the file attribute information corresponding to the file ID; and stores the second mapping between the file ID and the node address in the memory relationship, and store data block attribute information in the memory corresponding to the node address. The file path is the logical path of a file displayed to the user; actually, each file is divided into multiple data blocks, which are stored on multiple data...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/182
Inventor 李锐伍海君朱会灿邓大付邹永强董乘宇阙太富王磊杨绍鹏张书鑫赵大勇刘畅陈晓东张银锋
Owner TENCENT TECH (SHENZHEN) CO LTD