File system processing method and device, communication equipment, medium and system
By applying data set merging and compression processing in HDFS, dynamically allocating worker nodes is solved, and the problem of immutable data block size and number of replicas in HDFS is achieved, efficient resource utilization and system efficiency improvement.
Patent Information
- Application Number
- CN202410989401.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Hadoop Distributed File System (HDFS) has immutable data block size and number of replicas when read and write, resulting in waste of resources.
By applying preset rules between client devices and manager nodes, the data sets are merged or not merged, the number of interactions is reduced, and data block compression and merging is performed when writing, and the worker nodes are dynamically allocated to achieve variable Replica and data migration.
It effectively avoids resource waste, improves the resource utilization rate of worker nodes, enhances system processing efficiency, supports flexible write operations, and realizes the reasonable allocation of worker node resources.
Smart Images

Figure CN118939193A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of big data, and in particular to a file system processing method, device, communication equipment, medium and system. Background Art
[0002] In recent years, with the development of big data, Hadoop Distributed File System (HDFS) has been used to store and manage large-scale data sets due to its high fault tolerance and throughput. However, in related technologies, the data block size and number of replicas (worker nodes) of HDFS during reading and writing are immutable, that is, only a single data block size and fixed replicas can be used for reading and writing operations, which results in a certain waste of resources. Summary of the invention
[0003] The present disclosure provides a file system processing method, apparatus, communication equipment, medium and system, which merge or not merge a first data set by utilizing a first preset rule to reduce the number of interactions between a client device and an administrator node, thereby avoiding waste of resources.
[0004] The first aspect of the present disclosure provides a file system processing method, which is executed by a client device, including: obtaining a first data set; merging or not merging the first data set according to a first preset rule to obtain a second data set; sending a first write request message to a manager node, the first write request message is used to request to write the second data set to a worker node; receiving a first response message sent by the manager node, the first response message is used to indicate whether the client device is allowed to write the second data set; determining a first data block based on the first response message and the second data set; and writing the first data block to the worker node.
[0005] In some embodiments of the present disclosure, writing a first data block to a worker node includes: sending a second write request message to a manager node, the second write request message being used to request writing the first data block to the worker node; receiving first indication message sent by the manager node, the first indication message being used to indicate an address of a first worker node, the first worker node allowing writing of the first data block; and writing the first data block to the first worker node.
[0006] In some embodiments of the present disclosure, writing the first data block to the first worker node includes: compressing the first data block to obtain a first compressed data block, and storing the compression process to a client device; based on the size of the first compressed data block, merging the first compressed data blocks to obtain a first merged data block; and writing the first merged data block to the first worker node.
[0007] In some embodiments of the present disclosure, a read request message is sent to a manager node, the read request message is used to request to read a third data set; second indication message sent by the manager node is received, the second indication message is used to indicate an address of a second worker node, and the second worker node is used to store a second data block generated by the third data set; based on the second indication message, a target worker node is determined from the second worker nodes; based on the address of the target worker node, the second data block is read; and based on the second data block, the third data set is read.
[0008] In a second aspect of the present disclosure, an embodiment of a file system processing method is proposed, which is executed by a manager node, including: receiving a first write request message sent by a client device, the first write request message is used to request to write a second data set to a worker node; based on the first write request message, determining a first response message, the first response message is used to indicate whether the client device is allowed to write the second data set; and sending the first response message to the client device.
[0009] In some embodiments of the present disclosure, the method also includes: receiving a second write request message sent by the client, the second write request message is used to request to write the first data block to the worker node; based on the write request, according to a second preset rule, determining the first worker node, the first worker node is allowed to write the first data block; sending a first indication message to the client device, the first indication message is used to indicate the address of the first worker node.
[0010] In some embodiments of the present disclosure, the method also includes: receiving read request information sent by a client device, the read request information is used to request to read a third data set; based on the read request information, determining to store a second worker node, the second worker node is used to store a second data block generated by the third data set; and sending second indication information to the client device, the second indication information is used to indicate an address of the second worker node.
[0011] In some embodiments of the present disclosure, the method further includes: determining a hot / cold level of the third data block according to an update time of the third data block stored in the third worker node; and performing data migration on the third data block based on the hot / cold level.
[0012] The third aspect embodiment of the present disclosure proposes a client device, which includes a transceiver module, a processing module and a transceiver module, the processing module is used to obtain a first data set; merge or not merge the first data set according to a first preset rule to obtain a second data set; determine a first data block based on a first response information and the second data set; write the first data block to a worker node; the transceiver module is used to send a first write request information to a manager node, the first write request information is used to request to write the second data set to the worker node; receive a first response information sent by the manager node, the first response information is used to indicate whether the client device is allowed to write the second data set.
[0013] The fourth aspect embodiment of the present disclosure proposes a manager node, which includes a transceiver module and a processing module. The transceiver module is used to receive a first write request information sent by a client device, and the first write request information is used to request to write a second data set to a worker node; send a first response information to the client device; the processing module is used to determine a first response information based on the first write request information, and the first response information is used to indicate whether the client device is allowed to write the second data set.
[0014] The fifth aspect embodiment of the present disclosure proposes a communication device, including: a transceiver; a memory; a processor, which is connected to the transceiver and the memory respectively, and is configured to control the wireless signal reception and transmission of the transceiver by executing computer-executable instructions on the memory, and is capable of executing the method described in the first or second aspect embodiment of the present disclosure.
[0015] The sixth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first or second aspect embodiment of the present disclosure.
[0016] The seventh aspect embodiment of the present disclosure proposes a communication system, including a client device and a manager node, wherein the client device is used to execute the method described in the first aspect embodiment of the present disclosure; the manager node is used to execute the method described in the second aspect embodiment of the present disclosure.
[0017] In summary, the file system processing method proposed in the present disclosure includes: obtaining a first data set; merging or not merging the first data set according to a first preset rule to obtain a second data set; sending a first write request message to a manager node, the first write request message is used to request to write the second data set to a worker node; receiving a first response message sent by the manager node, the first response message is used to indicate whether the client device is allowed to write the second data set; determining a first data block based on the first response message and the second data set; and writing the first data block to the worker node. The method disclosed in the present disclosure merges or not merges the first data set by utilizing the first preset rule to reduce the number of interactions between the client device and the manager node, thereby avoiding waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0019] Figure 1 An interactive diagram of a file system processing method provided by an embodiment of the present disclosure;
[0020] Figure 2 A flowchart of a file system processing method provided by an embodiment of the present disclosure;
[0021] Figure 3 A flowchart of another file system processing method provided by an embodiment of the present disclosure;
[0022] Figure 4 A flowchart of another file system processing method provided by an embodiment of the present disclosure;
[0023] Figure 5 A flowchart of another file system processing method provided by an embodiment of the present disclosure;
[0024] Figure 6 A system interaction diagram of a file system processing method provided by an embodiment of the present disclosure;
[0025] Figure 7 An example diagram of a file system processing method provided by an embodiment of the present disclosure;
[0026] Figure 8 An example diagram of another file system processing method provided by an embodiment of the present disclosure;
[0027] Fig. 9 An example diagram of another file system processing method provided by an embodiment of the present disclosure;
[0028] Fig.10An example diagram of another file system processing method provided by an embodiment of the present disclosure;
[0029] Fig.11 An example diagram of a file system processing method provided by an embodiment of the present disclosure;
[0030] Fig.12 A schematic diagram of the structure of a client device provided in an embodiment of the present disclosure;
[0031] Fig.13 A schematic diagram of the structure of a manager node provided in an embodiment of the present disclosure;
[0032] Fig.14 A schematic diagram of the structure of a communication device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions.
[0034] In recent years, with the development of big data, Hadoop Distributed File System (HDFS) has been used to store and manage large-scale data sets due to its high fault tolerance and throughput. However, in related technologies, the data block size and number of replicas (worker nodes) of HDFS during reading and writing are immutable, that is, only a single data block size and fixed replicas can be used for reading and writing operations, which results in a certain waste of resources.
[0035] The method disclosed in the present invention combines or does not combine the first data set by utilizing the first preset rule, so as to reduce the number of interactions between the client device and the manager node, thereby avoiding waste of resources.
[0036] The file system processing method provided by the present application is described in detail below with reference to the accompanying drawings.
[0037] Figure 1 The following is an interactive diagram of a file system processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the file system processing method includes steps 101-119.
[0038] Step 101: A client device obtains a first data set.
[0039] In some embodiments, the first data set may be the first or multiple IO files written by a user, or may be an IO file obtained from other devices, which is not limited in the present disclosure.
[0040] In some embodiments, the files in the first data set may be from devices with the same network address or from devices with different network addresses, which is not limited in the present disclosure.
[0041] Step 102: The client device merges or does not merge the first data set according to a first preset rule to obtain a second data set.
[0042] In some embodiments, the first preset rule is used to determine whether multiple first data sets can be merged.
[0043] In some embodiments, the first preset rule is, for example, that first data sets from the same network address can be merged; for example, that first data sets from the same network address or adjacent network addresses can be merged. The present disclosure does not limit the specific content of the first preset rule.
[0044] In some embodiments, a data set obtained by merging the first data sets and the first data set that has not been merged may be determined as the second data set.
[0045] Step 103: The client device sends a first write request message to the manager node.
[0046] In some embodiments, the client device sends a first write request message to the manager node to request to write the second data set to the worker node.
[0047] In other words, the first write request information is used to request to write the second data set to the worker node.
[0048] In some embodiments, the first write request information may include content information of the second data set, permission information of the second data set, etc., which is not limited in the present disclosure.
[0049] Step 104: The manager node determines first response information based on the first write request information.
[0050] In some embodiments, the manager node determines, based on the first write request information, whether the second data set corresponding to the first write request information is allowed to be written to the worker node, thereby determining the first response information.
[0051] In other words, the first response information is used to indicate whether the client device is allowed to write the second data set. In step 105, the manager node sends the first response information to the client device.
[0052] For example, the manager node can determine whether the second data set has been stored in the worker node through the first write request information. If it has been stored, it can be determined that the first response information is that the second data set has been stored in the worker node and the write request is rejected. If it has not been stored, it can be determined that the first response information is that the write request is approved.
[0053] For example, the manager node can determine whether the second data set has the permission to be written to the worker node through the first write request information. If it has the permission to be written, the first response information can be determined to be that the write request is approved; if it does not have the permission to be written, the first response information can be determined to be that the write request is rejected.
[0054] Step 105: The manager node sends a first response message to the client device.
[0055] In some embodiments, the manager node sends a first response message to the client device to indicate whether the client device can write the second data set to the worker node.
[0056] In other words, the first response information is used to indicate whether the client device is allowed to write the second data set.
[0057] Step 106: The client device sends a second write request message to the manager node.
[0058] In some embodiments, since the second data set itself contains a large amount of data information, it needs to be divided into different first data blocks for writing to reduce the pressure of data transmission during a single write, wherein multiple first data blocks divided from the same second data set can be written into the same or different worker nodes.
[0059] It should be understood that the sizes of different first data blocks may be the same or different, and the present disclosure is not limited to this.
[0060] In some embodiments, the client device sends a second write request message to the manager node to request to write the first data block to the worker node.
[0061] In other words, the second write request information is used to request to write the first data block to the worker node.
[0062] In some embodiments, the second request information may include the first data block.
[0063] Step 107: The manager node determines the first worker node based on the second write request information and according to a second preset rule.
[0064] In some embodiments, there are multiple worker nodes in the HDFS system, and the manager node determines one or more first worker nodes from the multiple worker nodes based on the second write request information and according to a second preset rule.
[0065] In other words, the first worker node is the worker node that is allowed to write the first data block.
[0066] In some embodiments, the second preset rule is, for example, determining the first worker node from one or more worker nodes storing data blocks of the same type based on the data type of the first data block; for example, determining the first worker node from one or more worker nodes storing data blocks with the same network address based on the network address of the first data block. The present disclosure does not limit the specific content of the second preset rule.
[0067] In some embodiments, the manager node may also determine the number of first worker nodes according to a second preset rule, and the number of worker nodes used to store different first data blocks may be the same or different, so that the HDFS system can support variable copies.
[0068] Step 108: The manager node sends first indication information to the client device.
[0069] In some embodiments, the manager node sends a first indication message to the client device so that the client device obtains an address of a first worker node for storing and writing a first data block.
[0070] In other words, the first indication information is used to indicate the address of the first worker node.
[0071] Step 109: The client device compresses the first data block to obtain a first compressed data block.
[0072] In some embodiments, the client device compresses the first data block to reduce the node space occupied by the first data block when writing to the first worker node.
[0073] In some embodiments, the compression process is, for example, data compression processing using an LZ4 algorithm, a ZLIB algorithm, etc., which is not limited in the present disclosure.
[0074] Step 110: The client device stores the compression process in the client device.
[0075] In some embodiments, the client device stores the compression process to the client device, so that when the client device reads, the compressed data can be decompressed according to the compression process to obtain the original data.
[0076] Step 111: The client device merges the first compressed data blocks based on the size of the first compressed data blocks to obtain a first merged data block.
[0077] In some embodiments, the first compressed data blocks can be merged according to the data block size set by the client device to reduce the space occupied by the data blocks on the worker node.
[0078] In some embodiments, the first merged data block may be a merged first compressed data block, or may be a first compressed data block that does not meet the merge requirement and has not been merged.
[0079] For example, in the HDFS system, when the size of the first compressed data block is smaller than the specified size (128M), the compressed data block needs to be filled (it can be filled with meaningless data) to the specified size before it can be written to the worker node, which will cause a certain amount of space resource waste. Taking "setting the compressed data block to be written to the worker node with a size of 128MB" as an example, when the first compressed data block #1 is 99MB, the first compressed data block #2 is 127MB, the first compressed data block #3 is 26MB, and the first compressed data block #4 is 2MB, the first compressed data block #1, the first compressed data block #2 and the first compressed data block #4 can be merged into the first merged data block #1, thereby avoiding resource waste, and the first compressed data block #3 is determined as the first merged data block #2.
[0080] Step 112, the client device writes the first merged data block to the first worker node.
[0081] In some embodiments, when the manager node allocates only one first worker node to the client device, the client device can directly write the first merged data block to the first worker node; when the manager node allocates multiple first worker nodes to the client device, the client device can write the first merged data to any one of the multiple first nodes (for example, to the nearest first node), and the first node can write the first merged data to other first nodes in sequence.
[0082] Step 113: The client device sends a read request message to the manager node.
[0083] In some embodiments, the client device sends a read request message to the manager node to request to read the third data set.
[0084] It should be understood that the third data set can be generated by splitting, decompressing, merging, and other processes of one or more second data blocks stored by the manager node, wherein the second data block can be any data block stored by the manager node.
[0085] In other words, the read request information is used to request to read the third data set.
[0086] Step 114: The manager node determines a second worker node based on the read request information.
[0087] In some embodiments, the manager determines a second worker node based on the read request information, wherein the second worker node is used to store a second data block generated by the third data set, thereby sending a storage address of the second data set to the client device.
[0088] Step 115: The manager node sends second indication information to the client device.
[0089] In some embodiments, second indication information is sent to the client device to indicate the address of the second worker node, so that the client device can read the second data block.
[0090] In other words, the second indication information is used to indicate the address of the second worker node.
[0091] Step 116: The client device determines a target worker node from the second worker nodes based on the second indication information.
[0092] In some embodiments, when the same second data block is only stored in one second worker node, the client device may determine the second worker node as the target worker node.
[0093] In some embodiments, when the same second data block is stored in multiple second worker nodes (i.e., there is at least one copy of the second data block, for example, the second worker node #1 and the second worker node #2 both store the second data block #1), the client device can select any one from the multiple second worker nodes and determine it as the target worker node (for example, the second worker node closest to the client device is determined as the target worker node).
[0094] Step 117, the client device reads the second data block based on the address of the target worker node.
[0095] In some embodiments, the client device reads the acquired data block from the target worker node based on the address of the target worker node.
[0096] Step 118: The client device reads the third data set based on the second data block.
[0097] In some embodiments, the client device performs steps such as merging and decompression based on the read second data block, thereby reading the third data set.
[0098] Step 119: The manager node determines the hotness or coldness level of the third data block according to the update time of the third data block stored in the third worker node.
[0099] In some embodiments, the manager node may determine the hot / cold level of the third data block according to the update time of the third data block stored in the third worker node, thereby performing data migration on the third data block to avoid wasting resources.
[0100] It should be understood that the third worker node can be any worker node in the HDFS system, and the third data block can be any data block stored in the third worker node.
[0101] In some embodiments, the hot / cold level of the third data block can be determined by a preset threshold. For example, when the update time of the third data block is greater than ten days, the third data block can be determined as a cold data block; when the update time of the third data block is less than ten days and greater than five days, the data block can be determined as a relatively cold data block. The present disclosure does not limit the threshold set when determining the hot / cold level.
[0102] Step 120: The manager node migrates data on the third data block based on the hot and cold levels.
[0103] In some embodiments, the manager node can migrate the third data block of the cold level to a worker node that is farther away from the client device, and migrate the third data block of the hot level to a worker node that is closer to the client device, so as to reasonably allocate the space resources of the worker nodes, but not limited to this, and the present disclosure does not limit the rules for data migration.
[0104] In summary, the file system processing method proposed in the present disclosure includes: the client device obtains a first data set; the client device merges or does not merge the first data set according to a first preset rule to obtain a second data set; the client device sends a first write request message to the manager node; the manager node determines a first response message based on the first write request message; the manager node sends the first response message to the client device; the client device sends a second write request message to the manager node; the manager node determines a first worker node according to a second preset rule based on the second write request message; the manager node sends a first indication message to the client device; the client device compresses the first data block to obtain a first compressed data block; the client device stores the compression process to the client device; the client The device merges the first compressed data blocks based on the size of the first compressed data blocks to obtain a first merged data block; the client device writes the first merged data block to the first worker node; the client device sends a read request message to the manager node; the manager node determines the second worker node based on the read request message; the manager node sends a second indication message to the client device; the client device determines the target worker node from the second worker nodes based on the second indication message; the client device reads the second data block based on the address of the target worker node; the client device reads the third data set based on the second data block; the manager node determines the hot / cold level of the third data block based on the update time of the third data block stored in the third worker node; the manager node migrates data on the third data block based on the hot / cold level. The method disclosed in the present invention reduces the interaction between the client device and the manager node by merging data blocks, thereby avoiding waste of resources; improves the resource utilization rate of the worker node by compressing the data blocks, so that the worker node can store more data blocks; the manager node uses preset rules to allocate the same or different worker nodes to the data blocks to be written (that is, the copies of the system are variable), making the write operation of the client device more flexible; by determining the hot and cold levels of the data blocks, the data blocks in the worker nodes are automatically migrated, and the reasonable allocation of the worker node resources is achieved, thereby improving the processing efficiency of the system and further avoiding waste of resources.
[0105] Figure 2 The flowchart of a file system processing method proposed by the present disclosure is further shown, which is used for execution by a client device. Figure 2 As shown, the method comprises the following steps:
[0106] Step 201: Acquire a first data set.
[0107] In some embodiments, the principle of step 201 is similar to Figure 1 The step 101 shown is the same as that shown in FIG. 101 , and reference may be made to the relevant description of the embodiment shown in step 101 , which will not be repeated here.
[0108] Step 202: According to a first preset rule, the first data set is merged or not merged to obtain a second data set.
[0109] In some embodiments, the principle of step 202 is similar to Figure 1 The step 102 shown is the same, and the relevant description of the embodiment shown in step 102 may be referred to, which will not be repeated here.
[0110] Step 203: Send a first write request message to the manager node.
[0111] In some embodiments, the principle of step 203 is similar to Figure 1 The step 103 shown is the same, and the relevant description of the embodiment shown in step 103 may be referred to, which will not be repeated here.
[0112] Step 204: Receive the first response information sent by the manager node.
[0113] In some embodiments, the principle of step 204 is similar to Figure 1 The step 105 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 105, which will not be repeated here.
[0114] Step 205: Send a second write request message to the manager node.
[0115] In some embodiments, the principle of step 205 is similar to Figure 1 The step 106 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 106, which will not be repeated here.
[0116] Step 206: Receive first indication information sent by the manager node.
[0117] In some embodiments, the principle of step 206 is similar to Figure 1 The step 108 shown is the same as that shown, and reference may be made to the relevant description of the embodiment shown in step 108, which will not be repeated here.
[0118] Step 207: compress the first data block to obtain a first compressed data block.
[0119] In some embodiments, the principle of step 207 is similar to Figure 1 The step 109 shown is the same as that shown, and reference may be made to the relevant description of the embodiment shown in step 109, which will not be repeated here.
[0120] Step 208: Store the compression process in the client device.
[0121] In some embodiments, the principle of step 208 is similar to Figure 1 The step 110 shown is the same, and the relevant description of the embodiment shown in step 110 may be referred to, which will not be repeated here.
[0122] Step 209: Based on the size of the first compressed data blocks, merge the first compressed data blocks to obtain a first merged data block.
[0123] In some embodiments, the principle of step 209 is similar to Figure 1 The step 111 shown is the same, and the relevant description of the embodiment shown in step 111 may be referred to, which will not be repeated here.
[0124] Step 210, write the first merged data block to the first worker node.
[0125] In some embodiments, the principle of step 210 is similar to Figure 1 The step 112 shown is the same, and the relevant description of the embodiment shown in step 112 may be referred to, which will not be repeated here.
[0126] Step 211: Send a read request message to the manager node.
[0127] In some embodiments, the principle of step 211 is similar to Figure 1 The step 113 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 113, which will not be repeated here.
[0128] Step 212: Receive second indication information sent by the manager node.
[0129] In some embodiments, the principle of step 212 is similar to Figure 1 The step 115 shown is the same, and the relevant description of the embodiment shown in step 115 may be referred to, which will not be repeated here.
[0130] Step 213: Determine a target worker node from the second worker nodes based on the second indication information.
[0131] In some embodiments, the principle of step 213 is similar to Figure 1 The step 116 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 116, which will not be repeated here.
[0132] Step 214, read the second data block based on the address of the target worker node.
[0133] In some embodiments, the principle of step 214 is similar to Figure 1 The step 117 is the same as shown, and the relevant description of the embodiment shown in step 117 can be referred to, which will not be repeated here.
[0134] Step 215: read a third data set based on the second data block.
[0135] In some embodiments, the principle of step 215 is similar to Figure 1The step 118 shown is the same as that shown in step 117, and the relevant description of the embodiment shown in step 117 may be referred to, which will not be repeated here.
[0136] It should be understood that one or more of the above steps 207-209 and steps 211-215 are optional in some embodiments, and the present disclosure is not limited thereto.
[0137] In this implementation mode or example, unless there is any contradiction, each step can be independent, arbitrarily combined or exchanged in order, the optional methods or optional examples can be arbitrarily combined, and can be arbitrarily combined with any steps of other implementation modes or other examples.
[0138] Figure 3 The flowchart of a file system processing method proposed by the present disclosure is further shown, which is used for execution by a client device. Figure 3 As shown, the method comprises the following steps:
[0139] Step 301: Acquire a first data set.
[0140] In some embodiments, the principle of step 301 is similar to Figure 1 Step 101 and Figure 2 The step 201 shown is the same, and the relevant description of the embodiments shown in step 101 and step 201 may be referred to, and will not be repeated here.
[0141] Step 302: According to a first preset rule, the first data set is merged or not merged to obtain a second data set.
[0142] In some embodiments, the principle of step 302 is similar to Figure 1 Step 102 and Figure 2 The step 202 shown is the same, and the relevant description of the embodiments shown in step 102 and step 202 may be referred to, and will not be repeated here.
[0143] Step 303: Send a first write request message to the manager node.
[0144] In some embodiments, the principle of step 303 is similar to Figure 1 Step 103 and Figure 2 The step 203 shown is the same, and the relevant description of the embodiments shown in step 103 and step 203 may be referred to, and will not be repeated here.
[0145] Step 304: Receive the first response information sent by the manager node.
[0146] In some embodiments, the principle of step 304 is similar to Figure 1 Step 105 and Figure 2The step 204 is the same, and reference may be made to the relevant description of the embodiments shown in step 105 and step 204, which will not be repeated here.
[0147] Step 305: Determine a first data block based on the first response information and the second data set.
[0148] In some embodiments, the principle of step 305 is similar to Figure 1 Steps 106, 108, 109-111 and Figure 2 Steps 205-209 are the same, and reference may be made to the relevant description of the embodiments shown in the above steps, which will not be repeated here.
[0149] Step 306, write the first data block into the worker node.
[0150] In some embodiments, the principle of step 306 is similar to Figure 1 Step 112 and Figure 2 The step 210 shown is the same, and the relevant description of the embodiments shown in step 112 and step 210 may be referred to, and will not be repeated here.
[0151] Figure 4 The flowchart of a file system processing method proposed by the present disclosure is further shown, which is used for execution by a manager node. Figure 4 As shown, the method comprises the following steps:
[0152] Step 401: Receive first write request information sent by a client device.
[0153] In some embodiments, the principle of step 401 is similar to Figure 1 The step 103 shown is the same, and the relevant description of the embodiment shown in step 103 may be referred to, which will not be repeated here.
[0154] Step 402: Determine first response information based on the first write request information.
[0155] In some embodiments, the principle of step 402 is similar to Figure 1 The step 104 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 104, which will not be repeated here.
[0156] Step 403: Send first response information to the client device.
[0157] In some embodiments, the principle of step 403 is similar to Figure 1 The step 105 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 105, which will not be repeated here.
[0158] Step 404: Receive second write request information sent by the client.
[0159] In some embodiments, the principle of step 404 is similar to Figure 1 The step 106 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 106, which will not be repeated here.
[0160] Step 405: Based on the second write request information and according to a second preset rule, determine the first worker node.
[0161] In some embodiments, the principle of step 405 is similar to Figure 1 The step 107 shown is the same as that shown, and reference may be made to the relevant description of the embodiment shown in step 107, which will not be repeated here.
[0162] Step 406: Send first indication information to the client device.
[0163] In some embodiments, the principle of step 406 is similar to Figure 1 The step 108 shown is the same as that shown, and reference may be made to the relevant description of the embodiment shown in step 108, which will not be repeated here.
[0164] Step 407: receiving read request information sent by the client device.
[0165] In some embodiments, the principle of step 407 is similar to Figure 1 The step 113 shown is the same, and reference may be made to the relevant description of the embodiment shown in step 113, which will not be repeated here.
[0166] Step 408, determine the second worker node based on the read request information.
[0167] In some embodiments, the principle of step 408 is similar to Figure 1 The step 114 shown is the same, and the relevant description of the embodiment shown in step 114 may be referred to, and will not be repeated here.
[0168] Step 409: Send second indication information to the client device.
[0169] In some embodiments, the principle of step 409 is similar to Figure 1 The step 115 shown is the same, and the relevant description of the embodiment shown in step 115 may be referred to, which will not be repeated here.
[0170] Step 410, determining the hotness or coldness level of the third data block according to the update time of the third data block stored in the third worker node.
[0171] In some embodiments, the principle of step 210 is similar to Figure 1 The step 119 shown is the same as that shown, and the relevant description of the embodiment shown in step 119 may be referred to, which will not be repeated here.
[0172] Step 411: Migrate data on the third data block based on the cold and hot levels.
[0173] In some embodiments, the principle of step 211 is similar to Figure 1 The step 120 shown is the same as that shown, and reference may be made to the relevant description of the embodiment shown in step 120, which will not be repeated here.
[0174] It should be understood that one or more of the above steps 404-411 are optional in some embodiments, and the present disclosure is not limited to this.
[0175] In this implementation mode or example, unless there is any contradiction, each step can be independent, arbitrarily combined or exchanged in order, the optional methods or optional examples can be arbitrarily combined, and can be arbitrarily combined with any steps of other implementation modes or other examples.
[0176] Figure 5 The flowchart of a file system processing method proposed by the present disclosure is further shown, which is used for execution by a manager node. Figure 5 As shown, the method comprises the following steps:
[0177] Step 501: Receive first write request information sent by a client device.
[0178] In some embodiments, the principle of step 501 is similar to Figure 1 Step 103, Figure 4 The step 401 shown is the same, and the relevant description of the embodiments shown in step 103 and step 401 may be referred to, and will not be repeated here.
[0179] Step 502: Determine first response information based on the first write request information.
[0180] In some embodiments, the principle of step 502 is similar to Figure 1 Step 104, Figure 4 The step 402 shown is the same, and the relevant description of the embodiments shown in step 104 and step 402 may be referred to, and will not be repeated here.
[0181] Step 503: Send a first response letter to the client device.
[0182] In some embodiments, the principle of step 503 is similar to Figure 1 Step 105, Figure 4 The step 403 shown is the same, and reference may be made to the relevant description of the embodiments shown in step 105 and step 403, which will not be repeated here.
[0183] Figure 6 FIG. 1 is a schematic diagram of an interaction of a communication system according to an embodiment of the present disclosure. Figure 6 As shown, including:
[0184] Step 601: A client device obtains a first data set.
[0185] Step 602: The client device merges or does not merge the first data set according to a first preset rule.
[0186] Step 603: The client device sends a first write request message to the manager node.
[0187] Step 604: The manager node determines first response information based on the first write request information.
[0188] Step 605: The manager node sends a first response message to the client device.
[0189] Step 606: The client device determines a first data block based on the first response information and the second data set.
[0190] Step 607: The client device writes the first data block to the worker node.
[0191] In some embodiments, the above method may also include the method described in the above embodiments of the client device side, the manager node side, etc., which will not be repeated here.
[0192] The following is an exemplary description of the disclosed method.
[0193] The file system processing method provided by this method can be applied to the HDFS system and is compatible with reading, writing, and migrating data blocks of specifications such as 1MB, 4MB, 64MB, and 128MB.
[0194] Writing process design:
[0195] like Figure 7 As shown, the writing process steps are detailed:
[0196] 1. Assume that the write IO size is 300MB file, divided into three blocks, and there are three data nodes DN1, DN2, and DN3.
[0197] 2. HDFS Client (and the above client devices) puts the IO to be written into the write queue first. If multiple IOs can be merged, the merged IOs are processed and the write queue is refreshed. The information of the original multiple IOs is recorded in the smeta file to facilitate the recovery of the source block data. If they cannot be merged, no operation is performed. Each time an IO to be written is taken from the write queue, a file upload request is initiated to the NameNode (the above manager node).
[0198] 3. NameNode checks whether the file already exists, checks permissions, responds, and informs HDFS Client that the file can be uploaded.
[0199] 4. The HDFS Client splits the blocks into 128MB and requests to upload a block1 after the split.
[0200] 5. NameNode returns the list of DataNodes (i.e. the worker nodes mentioned above) assigned to block1.
[0201] 6. HDFS Client requests the nearest DataNode (DN1) to establish a block transmission channel based on the returned DataNode list. DN1 requests DN2 to establish a channel, and DN2 requests DN3 to establish a channel.
[0202] 7. After the three DataNodes successfully establish the channel, they respond to the HDFS Client one by one.
[0203] 8. HDFS Client compresses block1 and sends the compressed data in a packet structure to DataNode1. After DataNode1 completes writing, it sends it to DataNode2, and so on until DataNode3 completes writing. Each time DataNode1 transmits a packet, it puts it into a response queue and waits for a response.
[0204] 9. After block1 is transferred, the HDFS Client requests the NameNode for the write location (DataNode list) of block2 again, and repeats steps 3 to 7.
[0205] In the write process, the Client will compress and pre-process the data at the block granularity, and then store the processed compressed data on the DataNode and return it.
[0206] Reading process design:
[0207] like Figure 8 As shown, the reading process steps are detailed:
[0208] 1. Assume that the write IO size is 300MB file, divided into three blocks, and there are three data nodes DN1, DN2, and DN3.
[0209] 2. HDFS Client sends a read request to NameNode.
[0210] 3. NameNode queries the metadata to find the DataNode address where the file blocks (block1, block2, block3) are located and returns it to the HDFS Client.
[0211] 4. Select a DataNode (based on the principle of proximity) to send a read request, such as reading block 1 of DataNode 1. HDFSClient receives data in packets and writes it to the target file.
[0212] 5. Repeat steps 3 to 4 until all blocks are read, merge the data written into the target file, and then decompress and restore the original block data according to the smeta metadata.
[0213] The HDFS Client initiates a read request, queries the NameNode for the DataNode where the block is located, and sends a read request to these data nodes. After reading the data, it is decompressed on the Client side.
[0214] The principle of mutable copy is as follows:
[0215] like Fig. 9 As shown, the number of replicas of different blocks can be the same or different.
[0216] Support data migration design:
[0217] On the client side, data is divided into four levels according to the update time of the block: hot, relatively hot, relatively cold, and cold.
[0218] The following is a detailed classification of hot and cold data, and the values are all for reference.
[0219] Hot: epoch<10
[0220] Hotter: 10 <epoch<50
[0221] Cooler: 10000 <epoch<20000
[0222] Cold: 20000 <epoch
[0223] Cold and cold data can be migrated to farther or newly added data nodes to improve data reading performance. Epoch is a custom parameter used to indicate the hot and cold data.
[0224] When the replica field is added and recorded in the metadata, when each block is written, the NameNode will select the DataNode based on the replica. Enabling this write strategy can reduce communication between nodes.
[0225] like Fig.10 As shown in the figure, when a new DataNode is added to the cluster, the cluster automatically triggers data migration and migrates the cooler and cold data to the new node. At the same time, the DataNode can be used normally as a replica node.
[0226] Data preprocessing design:
[0227] like Fig.11 As shown in the figure, a new write queue is added on the client side. When the block size is set to 128MB (default), all blocks in the queue are retrieved, small blocks are merged into blocks no larger than 128MB, the information of the original multiple blocks is recorded in the smeta file, and the write queue is updated. After the read request reads the block from the DataNode, the block before the merge is restored according to the smeta file on the client side and returned to the user.
[0228] The embodiments of the present disclosure also propose a device for implementing any of the above methods, for example, a device is proposed, the above device includes a unit or module for implementing each step performed by the client device in any of the above methods. For example, another device is also proposed, including a unit or module for implementing each step performed by the manager node in any of the above methods.
[0229] Fig.12 Schematic diagram of the structure of the client device proposed in the embodiment of the present disclosure. Fig.12 As shown, the client device 1200 includes: a transceiver module 1210 and a processing module 1220. In some embodiments, the transceiver module 1210 is used to send a first write request message to the manager node, the first write request message is used to request to write a second data set to the worker node; receive a first response message sent by the manager node, the first response message is used to indicate whether the client device is allowed to write the second data set. Optionally. The transceiver module 1210 is used to execute at least one of the communication steps (such as step 103, step 105, step 106, step 108, step 113, step 115, but not limited to this) of sending and / or receiving performed by the client device 1200 in any of the above methods, which will not be repeated here. The processing module 1220 is used to obtain a first data set; merge or not merge the first data set according to a first preset rule to obtain a second data set; determine a first data block based on the first response message and the second data set; and write the first data block to the worker node. The processing module 1220 is used to execute at least one of the communication steps (such as step 106, step 107, but not limited to these) executed by the client device 1200 in any of the above methods except for the sending and / or receiving communication steps, which will not be repeated here.
[0230] Fig.13 Schematic diagram of the structure of the manager node proposed in the embodiment of the present disclosure. Fig.13As shown, the manager node 1300 includes: a transceiver module 1310 and a processing module 1320. In some embodiments, the transceiver module 1310 is used to receive a first write request message sent by a client device, and the first write request message is used to request to write a second data set to a worker node; and send a first response message to the client device. Optionally, the transceiver module 1310 is used to execute at least one of the communication steps (such as step 103, step 105, step 106, step 108, step 113, step 115, but not limited to this) of sending and / or receiving performed by the manager node 1300 in any of the above methods, which will not be repeated here. The processing module 1320 is used to determine the first response message based on the first write request message, and the first response message is used to indicate whether the client device is allowed to write the second data set. Optionally, the processing module is also used to execute at least one of the other communication steps (such as step 104, step 107, step 114, step 119, step 120, but not limited to this) other than sending and / or receiving, which will not be repeated here.
[0231] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0232] like Fig.14 As shown, the communication device 1400 includes one or more processors 1401. The processor 1401 may be a general-purpose processor or a dedicated processor, for example, a baseband processor or a central processing unit. The baseband processor may be used to process the communication protocol and the communication data, and the central processing unit may be used to control the communication device (such as a base station, a baseband chip, a terminal device, a terminal device chip, a DU or a CU, etc.), execute a program, and process the data of the program. The processor 1401 is used to call instructions so that the communication device 1400 executes any of the above methods.
[0233] In some embodiments, the communication device 1400 further includes one or more memories 1402 for storing instructions. Optionally, all or part of the memory 1402 may also be outside the communication device 1400.
[0234] In some embodiments, the communication device 1400 further includes one or more transceivers 1403. When the communication device 1400 includes one or more transceivers 1403, the communication steps such as sending and receiving in the above method are performed by the transceiver 1403, and the other steps are performed by the processor 1401.
[0235] In some embodiments, the transceiver may include a receiver and a transmitter, and the receiver and the transmitter may be separate or integrated. Optionally, the terms such as transceiver, transceiver unit, transceiver, transceiver circuit, etc. may be replaced with each other, the terms such as transmitter, transmission unit, transmitter, transmission circuit, etc. may be replaced with each other, and the terms such as receiver, receiving unit, receiver, receiving circuit, etc. may be replaced with each other.
[0236] Optionally, the communication device 1400 further includes one or more interface circuits 1404, which are connected to the memory 1402. The interface circuit 1404 can be used to receive signals from the memory 1402 or other devices, and can be used to send signals to the memory 1402 or other devices. For example, the interface circuit 1404 can read instructions stored in the memory 1402 and send the instructions to the processor 1401.
[0237] The communication device 1400 described in the above embodiments may be a network device or a terminal, but the scope of the communication device 1400 described in the present disclosure is not limited thereto, and the structure of the communication device 1400 may not be limited thereto. Fig.14 The communication device may be an independent device or may be part of a larger device. For example, the communication device may be: 1) an independent integrated circuit IC, or a chip, or a chip system or subsystem; (2) a collection of one or more ICs, optionally, the above IC collection may also include a storage component for storing data and programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal device, an intelligent terminal device, a cellular phone, a wireless device, a handheld device, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) others, etc.
[0238] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the file system processing method described in the above embodiments of the present disclosure.
[0239] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0240] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0241] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0242] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0243] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0244] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0245] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0246] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A file system processing method, characterized in that: The method is performed by a client device and includes: Obtaining a first data set; According to a first preset rule, merging or not merging the first data set to obtain a second data set; Sending first write request information to the manager node, where the first write request information is used to request writing a second data set to the worker node; receiving first response information sent by the manager node, where the first response information is used to indicate whether the client device is allowed to write a second data set; Determine a first data block based on the first response information and the second data set; Write the first data block to the worker node.
2. The method according to claim 1, characterized in that Writing the first data block to the worker node comprises: Sending a second write request message to the manager node, where the second write request message is used to request writing a first data block to the worker node; receiving first indication information sent by the manager node, where the first indication information is used to indicate an address of a first worker node, and the first worker node is allowed to write the first data block; The first data block is written to the first worker node.
3. The method according to claim 2, characterized in that Writing the first data block to the first worker node comprises: compressing the first data block to obtain a first compressed data block, storing the compression process in the client device; Based on the size of the first compressed data blocks, merge the first compressed data blocks to obtain a first merged data block; The first merged data block is written to the first worker node.
4. The method according to claim 1, characterized in that: The method further comprises: Sending a read request message to the manager node, where the read request message is used to request to read a third data set; receiving second indication information sent by the manager node, where the second indication information is used to indicate an address of a second worker node, where the second worker node is used to store a second data block generated by the third data set; Based on the second indication information, determining a target worker node from the second worker nodes; Based on the address of the target worker node, reading the second data block; Based on the second data block, the third data set is read.
5. A file system processing method, characterized in that: The method is executed by a manager node, and includes: Receiving first write request information sent by a client device, wherein the first write request information is used to request writing a second data set to a worker node; Determine first response information based on the first write request information, where the first response information is used to indicate whether the client device is allowed to write the second data set; The first response information is sent to the client device.
6. The method according to claim 5, characterized in that The method further comprises: receiving second write request information sent by the client, where the second write request information is used to request writing a first data block to the worker node; Based on the write request, determining a first worker node according to a second preset rule, the first worker node being allowed to write the first data block; Sending first indication information to the client device, where the first indication information is used to indicate an address of a first worker node.
7. The method according to claim 5, characterized in that The method further comprises: receiving a read request message sent by the client device, wherein the read request message is used to request to read a third data set; Based on the read request information, determine to store a second worker node, where the second worker node is used to store a second data block generated by the third data set; Sending second indication information to the client device, where the second indication information is used to indicate an address of the second worker node.
8. The method according to claim 5, characterized in that The method further comprises: Determining a hot / cold level of the third data block according to an update time of the third data block stored in the third worker node; Based on the hot / cold levels, data migration is performed on the third data block.
9. A client device, characterized in that: Including processing module and transceiver module, The processing module is used to obtain a first data set; and merge or not merge the first data set according to a first preset rule to obtain a second data set; Determine a first data block based on the first response information and the second data set; Writing a first data block to the worker node; The transceiver module is used to send a first write request message to the manager node, wherein the first write request message is used to request to write a second data set to the worker node; and receive a first response message sent by the manager node, wherein the first response message is used to indicate whether the client device is allowed to write the second data set.
10. A manager node, characterized in that: Including transceiver module and processing module, The transceiver module is used to receive first write request information sent by a client device, where the first write request information is used to request to write a second data set to a worker node; and send the first response information to the client device; The processing module is used to determine first response information based on the first write request information, where the first response information is used to indicate whether the client device is allowed to write the second data set.
11. A communication device, wherein: include: Transceiver; Memory; A processor is connected to the transceiver and the memory, respectively, and is configured to control the wireless signal reception and transmission of the transceiver by executing computer executable instructions on the memory, and can implement the method described in any one of claims 1-4 or 5-8.
12. A computer storage medium, wherein: The computer storage medium stores computer executable instructions; after the computer executable instructions are executed by the processor, the method described in any one of claims 1-4 or 5-8 can be implemented.
13. A communication system, characterized in that: include: A client device, configured to execute the method according to any one of claims 1 to 4; A manager node, configured to execute the method as claimed in any one of claims 5 to 8.
