An HDFS data landing method and device, a server and a storage medium
Patent Information
- Application Number
- CN202310073533.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-30
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2043-01-30
AI Technical Summary
当memstore的值达到一定的时候，就会形成一个个StoreFile(文件块)；然而，这种先写入WAL日志在写入数据的机制，导致在HDFS(HadoopDistributed File System，一种分布式文件系统)的存算分离模式下，HBase写入数据是WAL写入的性能瓶颈，致使HBase数据存入的速度不高
[0042] The present invention provides a method for writing HDFS data to disk, comprising: an HDFS server obtaining data to be stored sent by a client device; determining whether the data to be stored belongs to a target file block; wherein the target file block is a file block of HBase WAL log; if yes, adding the data to be stored to a flush queue and an acknowledgment queue; if no, adding the data to be stored to the flush queue, and adding the data to be stored to the acknowledgment queue after the target file block has been flushed;
Smart Images
Figure CN116382565B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to an HDFS data persistence method, apparatus, HDFS server, and computer-readable storage medium. Background Technology
[0002] HBase (a distributed, column-oriented open-source database) provides a high-concurrency and persistent log storage and replay mechanism through its Write-Ahead Log (WAL). Before each data write operation is performed, it is recorded in the WAL. When a client submits data to a RegionServer (an HBase component), it first writes the data to the WAL log. Only when the WAL log is successfully written is the client notified of a successful data submission. If writing to the WAL log fails, the client is notified of the submission failure; this is essentially the data persistence process.
[0003] In existing technologies, all Regions (the basic unit of data distribution and access in HBase) on a RegionServer share a single HLog (WAL log). A data commit is first written to the WAL log, and only after a successful write is it written to the memstore (write cache). When the memstore value reaches a certain level, it forms StoreFiles (file blocks). However, this mechanism of writing to the WAL log before writing data results in a performance bottleneck for HBase data writing in the HDFS (Hadoop Distributed File System) storage-compute separation mode, leading to slow HBase data storage speed.
[0004] Therefore, improving the speed of HBase data storage and enhancing user experience are urgent problems that need to be solved. Summary of the Invention
[0005] The purpose of this invention is to provide a method, apparatus, HDFS server, and computer-readable storage medium for writing HDFS data to disk, so as to improve the speed of HBase data storage and enhance user experience.
[0006] To solve the above technical problems, the present invention provides a method for writing HDFS data to disk, comprising:
[0007] The HDFS server obtains the data to be stored sent by the client device;
[0008] Determine whether the data to be stored belongs to the target file block; wherein, the target file block is a file block of HBase WAL log;
[0009] If so, the data to be stored is added to the flush queue and the response queue;
[0010] If not, the data to be stored is added to the flush queue, and after the target file block is flushed, the data to be stored is added to the response queue.
[0011] Optionally, determining whether the file block to be stored is a target file block includes:
[0012] Determine whether the identifier of the file block to which the data to be stored belongs is in the blocking queue;
[0013] If so, then the data to be stored is determined to belong to the target file block;
[0014] If not, then it is determined that the data to be stored does not belong to the target file block.
[0015] Optionally, the method further includes:
[0016] When creating a file, determine whether the file is the HBase WAL log;
[0017] If it is the HBase WAL log, then the file inode number corresponding to the source code file path of the file is added to the blocking queue; wherein, the identifier is the file block ID.
[0018] Optionally, determining whether the file is the HBase WAL log includes:
[0019] Based on the file format, determine whether the file is the HBase WAL log;
[0020] If it is the HBase WAL log, then the step of adding the file inode number corresponding to the source code file path of the file to the blocking queue is executed.
[0021] Optionally, determining whether the file is the HBase WAL log based on its format includes:
[0022] Determine whether the file format is the preset HBase WAL log format;
[0023] If the file is the preset HBase WAL log format, then the file is determined to be the HBase WAL log, and the step of adding the file inode number corresponding to the source code file path of the file to the blocking queue is executed.
[0024] Optionally, after adding the data to be stored to the flush queue and the response queue, the method further includes:
[0025] The data to be stored in the current data to be stored in the data to be stored is processed in segments and the successful data to be stored is checked in a round-robin manner according to the response return information corresponding to the data to be stored in the response queue; wherein, the data to be stored is any of the data to be stored.
[0026] Based on the successful flush offset, determine whether the data to be stored has been successfully flushed;
[0027] If the data is successfully downloaded, a successful storage message for the currently stored data is returned to the client device.
[0028] If the data refresh fails, a storage failure message for the currently pending data is returned to the client device.
[0029] Optionally, determining whether the data to be stored has been successfully flushed based on the successful flush offset includes:
[0030] After all the data to be stored has been flushed, determine whether the successful flush offset corresponding to the data to be stored is equal to the offset corresponding to the data to be stored.
[0031] If the offset is equal to the offset of the data to be stored, then the data to be stored has been successfully flushed.
[0032] If the offset is not equal to the offset of the data to be stored, then the data to be stored has failed to be flushed.
[0033] The present invention also provides an HDFS data persistence device, applied to an HDFS server, comprising:
[0034] The data acquisition module is used to acquire the data to be stored sent by the client device;
[0035] The data judgment module is used to determine whether the data to be stored belongs to the target file block; wherein, the target file block is a file block of HBase WAL log;
[0036] The log storage module is used to add the data to be stored to the flush queue and the response queue if it belongs to the target file block;
[0037] A conventional storage module is used to add the data to be stored to the flush queue if it does not belong to the target file block, and to add the data to be stored to the response queue after the target file block is flushed.
[0038] The present invention also provides an HDFS server, comprising:
[0039] Memory, used to store computer programs;
[0040] A processor, used to implement the steps of the HDFS data persistence method as described above when executing the computer program.
[0041] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the HDFS data persistence method described above.
[0042] The present invention provides a method for writing HDFS data to disk, comprising: an HDFS server obtaining data to be stored sent by a client device; determining whether the data to be stored belongs to a target file block; wherein the target file block is a file block of HBase WAL log; if yes, adding the data to be stored to a flush queue and an acknowledgment queue; if no, adding the data to be stored to the flush queue, and adding the data to be stored to the acknowledgment queue after the target file block has been flushed;
[0043] As can be seen, this invention identifies HBase WAL logs by determining whether the data to be stored belongs to the target file block. This allows the identified HBase WAL log data to be simultaneously added to both the flush queue and the response queue, enabling the HBase WAL log data to return response information during the flushing process. This achieves data flushing to disk in a direct response sequence based on the HBase WAL mechanism, improving the speed of HBase data storage and enhancing the user experience. Furthermore, this invention also provides an HDFS data flushing device, an HDFS server, and a computer-readable storage medium, which also possess the aforementioned beneficial effects. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0045] Figure 1 A flowchart illustrating an HDFS data persistence method provided in an embodiment of the present invention;
[0046] Figure 2 A flowchart illustrating another method for writing HDFS data to disk provided in an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of the WAL log identification process for another HDFS data persistence method provided in an embodiment of the present invention;
[0048] Figure 4 This is a schematic diagram illustrating the file block writing process of another HDFS data writing method provided in an embodiment of the present invention;
[0049] Figure 5 This is a schematic diagram of the ack fault tolerance process of another HDFS data persistence method provided in an embodiment of the present invention;
[0050] Figure 6 This is a structural block diagram of an HDFS data persistence device provided in an embodiment of the present invention;
[0051] Figure 7 This is a schematic diagram of the structure of an HDFS server provided in an embodiment of the present invention;
[0052] Figure 8 This is a schematic diagram of the specific structure of an HDFS server provided in an embodiment of the present invention;
[0053] Figure 9 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation
[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0055] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for writing HDFS data to disk, as provided in an embodiment of the present invention. The method may include:
[0056] Step 101: The HDFS server obtains the data to be stored sent by the client device.
[0057] HDFS (Hadoop Distributed File System) is a Hadoop distributed file system. HDFS is highly fault-tolerant and can be deployed on low-cost hardware. In this embodiment, the HDFS server provides highly reliable underlying storage support for HBase; that is, the HDFS server can be deployed with HBase.
[0058] Accordingly, the client device in this embodiment can be a device with the client program corresponding to the HDFS server installed. The data to be stored in this embodiment can be HBase data sent by the client device to the HDFS server, that is, data stored using HBase, such as data submitted by the client device to the HBase RegionServer.
[0059] Specifically, the method by which the processor in the HDFS server obtains the data to be stored sent by the client device in this step can be set by the designer according to the practical scenario and user needs. For example, it can be implemented in a way that is the same as or similar to the HBase data receiving method of the BlockReceiver layer (a class name used for data transmission in HDFS) in the existing technology. For example, the processor can obtain the data to be stored in a packet sent by the client device. As long as the processor in the HDFS server can obtain the data to be stored sent by the client device, this embodiment does not impose any restrictions on this.
[0060] Step 102: Determine whether the data to be stored belongs to the target file block; if yes, proceed to step 103; if no, proceed to step 104.
[0061] The target file block is the HBase WAL log file block. In this step, the processor determines whether the data to be stored belongs to the target file block and detects whether the acquired data to be stored is HBase WAL log data (i.e., WAL data), thereby enabling the BlockReceiver layer to identify HBase WAL logs.
[0062] Specifically, the method by which the processor in the HDFS server determines whether the data to be stored belongs to the target file block in this embodiment can be set by the designer according to the practical scenario and user needs. For example, the processor can determine whether the identifier of the file block to which the data to be stored belongs is in the blocking queue; if so, it is determined that the data to be stored belongs to the target file block, that is, the data to be stored is HBase WAL log data; if not, it is determined that the data to be stored does not belong to the target file block, that is, the data to be stored is HBase WAL log data. In other words, in this embodiment, the identifier of the file block corresponding to the HBase WAL log can be pre-placed in the blocking queue through the setting of the blocking queue to facilitate the subsequent identification of the HBase WAL log. The processor can also determine whether the identifier of the file block to which the data to be stored belongs is an identifier in a preset identifier table. This embodiment does not impose any restrictions on this.
[0063] Correspondingly, the specific selection of the identifier for the aforementioned file block can be set by the designer according to the practical scenario and user needs. For example, the identifier for the file block can be the file block ID (i.e., the block ID), which is the file inode number (ino number) corresponding to the source code (source, src) file path stored in the file block. In other words, in this embodiment, the file inode number corresponding to the file path (i.e., the src file path) of the HBase WAL log file can be added to the blocking queue in advance so that subsequent processing can identify the HBase WAL log data by judging whether the block ID of the file block to which the data to be stored belongs is in the blocking queue. For example, when the BlockReceiver layer is writing data to disk, it can detect the file block to which the packet belongs. If the block ID of the file block is in the blocking queue, it will proceed to step 103, and while the data of the packet is being flushed to the queue, it will also be added to the acknowledgment (ack) queue.
[0064] Accordingly, the HDFS data persistence method provided in this embodiment may also include a blocking queue configuration process. For example, when creating a file, the processor can determine whether the file is an HBase WAL log. If it is an HBase WAL log, the file inode number corresponding to the file's source code file path is added to the blocking queue. For example, since HBase WAL logs have a specific format in HDFS storage (i.e., a preset HBase WAL log format), in this step, the processor can determine whether the file is an HBase WAL log based on the file's format, and thus add the file inode number corresponding to the file's source code file path to the blocking queue when determining whether the file is an HBase WAL log. For example, the processor can determine whether the file's format is a preset HBase WAL log format; if it is a preset HBase WAL log format, the file is determined to be an HBase WAL log.
[0065] Step 103: Add the data to be stored to the flush queue and the response queue.
[0066] It is understandable that in this step, the processor in the HDFS server can determine that the data to be stored belongs to the target file block, that is, the data to be stored is HBase WAL log data, and then add the data to be stored to both the flush queue and the acknowledgment queue to achieve the ack return during the data flushing process. In other words, in this embodiment, the WAL data storage speed can be improved by directly returning ack, thereby improving the HBase data storage speed.
[0067] Specifically, the method by which the processor adds the data to be stored to the flush queue and the acknowledgment queue in this embodiment can be set by the designer according to the practical scenario and user needs. For example, the processor can directly add the content of the data to be stored to the flush queue and the acknowledgment queue, or it can add the identifier of the data to be stored (such as the block ID) to the flush queue and the acknowledgment queue. As long as it is ensured that the processor can achieve the ack return during the data flushing process after adding the data to be stored to the flush queue and the acknowledgment queue, this embodiment does not impose any restrictions on this.
[0068] Furthermore, in this embodiment, since the HBase WAL log data flushing process directly returns an ACK, the correctness of data write-to-disk can be guaranteed through the ACK fault tolerance mechanism. For example, during the HBase WAL log data flushing process, the processor can prevent the returned response information from being directly returned to the client device. Instead, it can check the successful offset based on the response information to determine and return the HBase WAL log data write-to-disk result to the client device. For example, the processor can perform segmented flushing of the currently pending data in the flushing queue and poll the successful flushing offset corresponding to the currently pending data in the response queue based on the response information of the currently pending data. Here, the currently pending data can be any pending data. Based on the successful flushing offset, it can be determined whether the currently pending data has been flushed successfully. If the flushing is successful, the storage success information of the currently pending data is returned to the client device; if the flushing fails, the storage failure information of the currently pending data is returned to the client device. The processor can also flush the currently pending data in the flush queue and check the successful flush offset of the current file corresponding to the currently pending data based on the response return information of the currently pending data in the response queue. After all the pending data corresponding to the current file has been flushed, it determines whether the current pending data has been flushed successfully based on the offset of the current file and the successful flush offset. If the flush is successful, it returns the storage success information of the current file to the client device; if the flush fails, it returns the storage failure information of the current file to the client device.
[0069] Step 104: Add the data to be stored to the flush queue, and after the target file block is flushed, add the data to be stored to the response queue.
[0070] Understandably, in this step, the processor in the HDFS server can add the data to be stored to the flush queue first when it is determined that the data to be stored does not belong to the target file block, that is, the data to be stored is not HBase WAL log data, and then add the data to the response queue after the target file block is flushed.
[0071] Specifically, this embodiment does not limit the specific method by which the processor adds the data to be stored to the flush queue and adds the data to the acknowledgment queue after the target file block is flushed. For example, the processor can adopt the same or similar method as the existing technology of flushing the data to disk first and then returning ack.
[0072] In this embodiment, the present invention identifies HBase WAL logs by determining whether the data to be stored belongs to the target file block; thereby, the identified HBase WAL log data is simultaneously added to the flush queue and the response queue, so that the HBase WAL log data can return response information when flushing to disk, realizing data flushing to disk in direct response order based on the HBase WAL mechanism, improving the speed of HBase data storage and enhancing the user experience.
[0073] Based on the above embodiments, this invention also provides another method for writing HDFS data to disk to ensure the correctness of data writing. For details, please refer to... Figure 2 Please refer to Figure 2 , Figure 2 A flowchart illustrating another method for writing HDFS data to disk, provided in an embodiment of the present invention. This method may include:
[0074] Step 201: The HDFS server obtains the data to be stored sent by the client device.
[0075] In this embodiment, the HDFS server provides highly reliable underlying storage support for HBase; that is, the HDFS server can be deployed with HBase. The client device in this embodiment can be a device with the client program corresponding to the HDFS server installed. The data to be stored in this embodiment can be HBase data sent by the client device to the HDFS server, i.e., data stored using HBase, such as data submitted by the client device to an HBase RegionServer.
[0076] Specifically, the method by which the processor in the HDFS server obtains the data to be stored sent by the client device in this step can be set by the designer according to the practical scenario and user needs. For example, it can be implemented in the same or similar way as the HBase data receiving method of the BlockReceiver layer (file block receiving layer) in the existing technology. For example, the processor can obtain the data to be stored in a file packet sent by the client device. As long as the processor in the HDFS server can obtain the data to be stored sent by the client device, this embodiment does not impose any restrictions on this.
[0077] Step 202: Determine whether the identifier of the file block to which the data to be stored belongs is in the blocking queue. If not, proceed to step 203; if yes, proceed to step 204.
[0078] The target file block is the HBase WAL log file block. In this step, the processor determines whether the identifier (e.g., block ID) of the file block to which the data to be stored belongs is in the blocking queue, and detects whether the acquired data to be stored is HBase WAL log data (i.e., WAL data). This enables the BlockReceiver layer to identify HBase WAL logs; for example, the BlockReceiver layer can recognize that HBase WAL logs have been added to the flush queue and return an ACK response: when the flush interface writes data to disk, it determines the file block to which the packet belongs, and if the block ID (e.g., block ID) is in the blocking queue, it checks whether the data is in the blocking queue. Figure 4 If the Block ID is in the blocking queue, then the data of that packet will be added to the acknowledgment queue at the same time as the data is flushed to the queue (e.g., ...). Figure 4 (The ack queue in the middle).
[0079] Specifically, the method by which the processor in the HDFS server determines whether the data to be stored belongs to the target file block in this embodiment can be set by the designer according to the practical scenario and user needs. For example, the processor can determine whether the block ID of the file block to which the data to be stored belongs is in the blocking queue; if so, the data to be stored is determined to be HBase WAL log data; if not, the data to be stored is HBase WAL log data. In other words, in this embodiment, the block ID of the file block corresponding to the HBase WAL log can be pre-placed in the blocking queue through the setting of the blocking queue to facilitate the subsequent identification of the HBase WAL log.
[0080] Correspondingly, the HDFS data persistence method provided in this embodiment may also include a blocking queue configuration process. For example, when creating a file, the processor can determine whether the file is an HBase WAL log. If it is an HBase WAL log, the file inode number corresponding to the file's source code file path is added to the blocking queue. For example, since HBase WAL logs have a specific format in HDFS storage (i.e., a preset HBase WAL log format), in this step, the processor can determine whether the file is an HBase WAL log based on the file's format, and thus add the file inode number corresponding to the file's source code file path to the blocking queue when determining whether the file is an HBase WAL log. For example, the processor can determine whether the file's format is a preset HBase WAL log format; if it is a preset HBase WAL log format, the file is determined to be an HBase WAL log.
[0081] For example, such as Figure 3 The HDFSRpcServer layer (HDFS Remote Procedure Call Protocol Service Layer) can recognize HBase WAL logs: When creating a file, the HDFS server's processor can pass in the src file path and split the src file path; for src file paths that conform to the WAL log format (i.e., the default HBase WAL log format), it indicates that HBase WAL log data is about to be stored, and the ino (i.e., file inode number) of the src file path is added to the blocking queue; the corresponding file status is returned to indicate that file permissions and other conditions allow the input stream to be opened and data to be stored; while for file paths that do not conform to the WAL log format, the corresponding file status can be returned directly, as in existing technologies.
[0082] Step 203: Add the data to be stored to the flush queue, and after the target file block is flushed, add the data to be stored to the response queue.
[0083] Understandably, in this step, the processor in the HDFS server can add the data to be stored to the flush queue first when it is determined that the data to be stored does not belong to the target file block, that is, the data to be stored is not HBase WAL log data, and then add the data to the response queue after the target file block is flushed.
[0084] Specifically, this embodiment does not limit the specific method by which the processor adds the data to be stored to the flush queue and adds the data to the acknowledgment queue after the target file block is flushed. For example, the processor can adopt the same or similar method as the existing technology of flushing the data to disk first and then returning ack.
[0085] Step 204: Add the data to be stored to the flush queue and the response queue.
[0086] It is understandable that in this step, the processor in the HDFS server can determine that the data to be stored belongs to the target file block, that is, the data to be stored is HBase WAL log data, and then add the data to be stored to both the flush queue and the acknowledgment queue to achieve the ack return during the data flushing process. In other words, in this embodiment, the WAL data storage speed can be improved by directly returning ack, thereby improving the HBase data storage speed.
[0087] Specifically, the method by which the processor adds the data to be stored to the flush queue and the acknowledgment queue in this embodiment can be set by the designer according to the practical scenario and user needs. For example, the processor can directly add the content of the data to be stored to the flush queue and the acknowledgment queue, or it can add the identifier of the data to be stored (such as the block ID) to the flush queue and the acknowledgment queue. As long as it is ensured that the processor can achieve the ack return during the data flushing process after adding the data to be stored to the flush queue and the acknowledgment queue, this embodiment does not impose any restrictions on this.
[0088] Since the HBase WAL log data flushing process directly returns an ACK, this embodiment can ensure the correctness of data write-to-disk by setting an ACK fault tolerance mechanism. For example, during the HBase WAL log data flushing process, the processor can prevent the returned response information from being directly returned to the client device. Instead, it can check the successful offset based on the response information to determine and return the HBase WAL log data write-to-disk result to the client device. For example, the processor can perform segmented flushing of the currently to-be-stored data in the flushing queue through steps 205 to 208, and poll the successful flushing offset corresponding to the currently to-be-stored data based on the response information of the currently to-be-stored data in the response queue; where the currently to-be-stored data is any data to be stored; based on the successful flushing offset, it can determine whether the currently to-be-stored data has been successfully flushed; if the flushing is successful, it can return the storage success information of the currently to-be-stored data to the client device; if the flushing fails, it can return the storage failure information of the currently to-be-stored data to the client device.
[0089] Correspondingly, in some other embodiments, the processor can also adopt other ACK fault tolerance mechanisms to ensure the correctness of data writing to disk. For example, it can flush the currently to-be-stored data in the flush queue and check the successful flush offset of the current file corresponding to the currently to-be-stored data based on the ACK return information corresponding to the currently to-be-stored data in the ACK queue. After all the to-be-stored data corresponding to the current file has been flushed, it can determine whether the currently to-be-stored data has been flushed successfully based on the offset corresponding to the current file and the successful flush offset. If the flush is successful, it returns the storage success information of the current file to the client device; if the flush fails, it returns the storage failure information of the current file to the client device.
[0090] Step 205: Perform segmented flushing of the current data to be stored in the flushing queue, and poll the successful flushing offset corresponding to the current data to be stored based on the response return information corresponding to the current data to be stored in the response queue.
[0091] It is understood that the data to be stored is any data to be stored. In this embodiment, the processor can perform segmented flushing of the data to be stored in the flushing queue to realize the segmented flushing of the HBase WAL log file. Thus, the offset of the successful flushing can be updated after each segment is flushed. When the finally updated offset is equal to the offset of the file itself, it is determined that the file has been successfully written to disk; otherwise, it fails and the client device is notified accordingly.
[0092] like Figure 5 As shown in this embodiment, since the HBase WAL log directly returns ack (i.e., response return information) during the data flushing process, the processor can perform segmented flushing of the currently to-be-stored data in the flushing queue; and during segmented flushing, based on the response return information corresponding to the currently to-be-stored data in the response queue, it starts polling to check the offset value of the currently to-be-stored data that has been successfully flushed, and records the latest value (i.e., the successful flushing offset).
[0093] Step 206: Based on the successful data drop offset, determine whether the data to be stored has been successfully dropped; if yes, proceed to step 207; if no, proceed to step 208.
[0094] Understandably, in this step, the processor can determine whether the current data to be stored has been successfully flushed based on the successful flush offset corresponding to the current data to be stored. For example, after all the data to be stored has been flushed, the processor can determine whether the successful flush offset corresponding to the current data to be stored is equal to the current offset corresponding to the current data to be stored. If it is equal to the current offset, the current data to be stored is determined to have been successfully flushed; if it is not equal to the current offset, the current data to be stored is determined to have failed to flush. Figure 5 As shown, after polling is complete, the processor can determine whether the latest value of the record corresponding to the data to be stored (i.e., the successful flush offset) is equal to the offset corresponding to the data to be stored. If it is equal to the offset corresponding to the data to be stored, the data to be stored is determined to have been successfully flushed. If it is not equal to the offset corresponding to the data to be stored, the data to be stored is determined to have failed to be flushed.
[0095] Step 207: Return a message to the client device indicating successful storage of the data to be stored.
[0096] In this step, the processor in the HDFS server can return a successful storage message to the client device when the data to be stored is successfully written to disk. This indicates that the client device has successfully written the data to disk.
[0097] Step 208: Return storage failure information for the current data to be stored to the client device.
[0098] Specifically, in this step, the processor in the HDFS server can return a storage failure message to the client device when the current data to be stored fails to be flushed. This indicates that the client device's data to be stored has failed to be written to disk.
[0099] In this embodiment, the present invention implements an ACK fault tolerance mechanism by segmenting the data to be stored in the data-to-disk queue and polling the successful data-to-disk offset corresponding to the data to be stored in the response queue based on the response return information corresponding to the data to be stored in the response queue, thereby further ensuring the correctness of data write-to-disk.
[0100] Based on the previous embodiment, this embodiment of the invention also provides another WAL log identification process for HDFS data persistence, so as to realize the identification of HBase WAL logs by the BlockReceiver layer through the configuration of file block IDs in the blocking queue. Specifically, the configuration process of file block IDs in the blocking queue may include:
[0101] Step 1: When creating a file, determine whether the file is an HBase WAL log.
[0102] Specifically, because HBase WAL logs have a specific format in HDFS storage (i.e., the default HBase WAL log format), in this step, the HDFS server processor can determine whether a file is an HBase WAL log based on its format when creating the file. Upon determining if a file is an HBase WAL log, the inode number (ino) corresponding to the file's source code file path (src file path) is added to a blocking queue. For example, the processor can check if the file format matches the default HBase WAL log format; if it does, the file is determined to be an HBase WAL log. For instance, the processor can pass the src file path when creating the file and split the src file path; for src file paths that conform to the WAL log format (i.e., the default HBase WAL log format), it indicates that HBase WAL log data is about to be stored, meaning the file corresponding to that src file path is an HBase WAL log.
[0103] Step 2: If it is an HBase WAL log, add the file inode number corresponding to the source code file path to the blocking queue.
[0104] Understandably, in this step, when the processor recognizes that the created file is an HBase WAL log, it can add the block ID (file block ID) of the file block belonging to the source code file path (i.e., HBase WAL log) that needs to be stored later to the blocking queue by adding the file inode number corresponding to the source code file path of the file to the blocking queue, thereby realizing the block ID configuration of the file block belonging to the HBase WAL log in the blocking queue.
[0105] Correspondingly, in this step, the processor can also return the file status of the created file to the client device. For example, if the file is an HBase WAL log, the processor can add the file inode number corresponding to the file's source code file path to the blocking queue and then return the corresponding file status to the client device to indicate that the file permissions and other conditions allow the input stream to be opened and data to be stored. If the file is not an HBase WAL log, the processor can directly return the corresponding file status as in existing technologies.
[0106] like Figure 3 The HDFSRpcServer layer can recognize HBase WAL logs: When creating a file, the HDFS server's processor can pass in the src file path and split the src file path; for file paths that conform to the WAL log format, it indicates that the data to be stored is HBase WAL log data, and the ino (i.e., file inode number) of the src file path is added to the blocking queue; the corresponding file status is returned to indicate that file permissions and other conditions allow the input stream to be opened and data to be stored; for file paths that do not conform to the WAL log format, the corresponding file status can be returned directly.
[0107] Corresponding to the above method embodiments, this invention also provides an HDFS data persistence device. The HDFS data persistence device described below and the HDFS data persistence method described above can be referred to each other.
[0108] Please refer to Figure 6 , Figure 6 This is a structural block diagram of an HDFS data persistence device provided in an embodiment of the present invention. The device is applied to an HDFS server and may include:
[0109] The data acquisition module 10 is used to acquire the data to be stored sent by the client device;
[0110] The data judgment module 20 is used to determine whether the data to be stored belongs to the target file block; wherein, the target file block is the file block of the HBase WAL log;
[0111] Log storage module 30 is used to add the data to be stored to the flush queue and the response queue if it belongs to the target file block;
[0112] The conventional storage module 40 is used to add the data to be stored to the flush queue if it does not belong to the target file block, and to add the data to be stored to the response queue after the target file block is flushed.
[0113] Optionally, the data judgment module 20 can be specifically used to determine whether the identifier of the file block to which the data to be stored belongs is in the blocking queue; if yes, then it is determined that the data to be stored belongs to the target file block; if no, then it is determined that the data to be stored does not belong to the target file block.
[0114] Optionally, the device may also include:
[0115] Create an identification module to determine whether a file is an HBase WAL log when the file is created;
[0116] The queue configuration module is used to add the file inode number corresponding to the source code file path of the HBase WAL log to the blocking queue; where the identifier is the file block ID.
[0117] Optionally, the creation identification module can be specifically used to determine whether a file is an HBase WAL log based on its format when creating a file; if it is an HBase WAL log, a start signal is sent to the queue configuration module.
[0118] Optionally, creating a recognition module may include:
[0119] The judgment submodule is used to determine whether the file format is the preset HBase WAL log format;
[0120] The determination submodule is used to determine that the file is an HBase WAL log if the default HBase WAL log format is used, and to send a start signal to the queue configuration module.
[0121] Optionally, the device may also include:
[0122] The flush polling module is used to flush the currently pending data in the flush queue in segments, and to poll the successful flush offset corresponding to the currently pending data based on the response return information of the currently pending data in the response queue; where the currently pending data can be any pending data.
[0123] The down-flush judgment module is used to determine whether the data to be stored has been successfully flushed based on the successful down-flush offset.
[0124] The success return module is used to return a successful storage message to the client device if the data is successfully stored.
[0125] The failure return module is used to return storage failure information of the current data to be stored to the client device if the data refresh fails.
[0126] Optionally, the flushing judgment module can be specifically used to determine whether the successful flushing offset corresponding to the current data to be stored is equal to the offset corresponding to the current data to be stored after all the data to be stored has been flushed; if it is equal to the offset corresponding to the current data to be stored, it is determined that the current data to be stored has been flushed successfully, and a start signal is sent to the success return module; if it is not equal to the offset corresponding to the current data to be stored, it is determined that the current data to be stored has failed to flush, and a start signal is sent to the failure return module.
[0127] In this embodiment, the data judgment module 20 determines whether the data to be stored belongs to the target file block, thereby recognizing the HBase WAL log. This allows the recognized HBase WAL log data to be simultaneously added to both the flush queue and the response queue, enabling the HBase WAL log data to return response information during flushing to disk. This achieves direct response-based data flushing to disk based on the HBase WAL mechanism, improving the speed of HBase data storage and enhancing the user experience.
[0128] Corresponding to the above method embodiments, this invention also provides an HDFS server. The HDFS server described below and the HDFS data persistence method described above can be referred to in correspondence.
[0129] Please refer to Figure 7 , Figure 7 This is a schematic diagram of an HDFS server provided in an embodiment of the present invention. The HDFS server may include:
[0130] Memory D1 is used to store computer programs;
[0131] Processor D2 is used to execute computer programs to implement the steps of the HDFS data persistence method provided in the above method embodiments.
[0132] For details, please refer to Figure 8 , Figure 8This is a schematic diagram illustrating the specific structure of an HDFS server provided in an embodiment of the present invention. The HDFS server 310 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing applications 342 or data 344. The memory 332 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the central processing unit 322 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the HDFS server 310.
[0133] The storage medium 330 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the storage medium 330 stores at least a program for implementing the following functions:
[0134] Retrieve data to be stored sent by the client device;
[0135] Determine whether the data to be stored belongs to the target file block; where the target file block is the file block of the HBase WAL log.
[0136] If so, the data to be stored will be added to the flush queue and the response queue;
[0137] If not, the data to be stored is added to the flush queue, and after the target file block is flushed, the data to be stored is added to the response queue.
[0138] In some embodiments, the storage medium 330 may also store a program for implementing the following functions:
[0139] When creating a file, check if the file is an HBase WAL log;
[0140] If it is an HBase WAL log, then the file inode number corresponding to the file's source code file path is added to the blocking queue.
[0141] In some embodiments, the storage medium 330 may also store a program for implementing the following functions:
[0142] The data to be stored in the current data queue is segmented and flushed down. Based on the response information corresponding to the data to be stored in the response queue, the successful flush offset corresponding to the data to be stored is checked in a round-robin fashion. The data to be stored can be any data to be stored.
[0143] Based on the successful flush offset, determine whether the data to be stored has been successfully flushed;
[0144] If the data is successfully stored, a message indicating successful storage of the data to be stored is returned to the client device.
[0145] If the data refresh fails, a storage failure message for the currently pending data will be returned to the client device.
[0146] The HDFS server 310 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341. Examples include Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0147] The steps in the HDFS data persistence method described above can be implemented by the structure of the HDFS server.
[0148] Corresponding to the above method embodiments, this invention also provides a computer-readable storage medium. The computer-readable storage medium described below can be referred to in conjunction with the HDFS data persistence method described above.
[0149] Please refer to Figure 9 , Figure 9 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of the present invention. The computer-readable storage medium 50 stores a computer program 51, which, when executed by a processor, implements the steps of the HDFS data persistence method provided in the above-described method embodiment.
[0150] When computer program 51 is executed by the processor, it can at least achieve the following:
[0151] Retrieve data to be stored sent by the client device;
[0152] Determine whether the data to be stored belongs to the target file block; where the target file block is the file block of the HBase WAL log.
[0153] If so, the data to be stored will be added to the flush queue and the response queue;
[0154] If not, the data to be stored is added to the flush queue, and after the target file block is flushed, the data to be stored is added to the response queue.
[0155] In some embodiments, when the computer program 51 is executed by the processor, it can also perform the following:
[0156] When creating a file, check if the file is an HBase WAL log;
[0157] If it is an HBase WAL log, then the file inode number corresponding to the file's source code file path is added to the blocking queue.
[0158] In some embodiments, when the computer program 51 is executed by the processor, it can also perform the following:
[0159] The data to be stored in the current data queue is segmented and flushed down. Based on the response information corresponding to the data to be stored in the response queue, the successful flush offset corresponding to the data to be stored is checked in a round-robin fashion. The data to be stored can be any data to be stored.
[0160] Based on the successful flush offset, determine whether the data to be stored has been successfully flushed;
[0161] If the data is successfully stored, a message indicating successful storage of the data to be stored is returned to the client device.
[0162] If the data refresh fails, a storage failure message for the currently pending data will be returned to the client device.
[0163] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The apparatus, HDFS server, and computer-readable storage medium disclosed in the embodiments are described simply because they correspond to the methods disclosed in the embodiments; relevant details can be found in the method section.
[0164] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0165] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0166] The present invention has provided a detailed description of an HDFS data persistence method, apparatus, HDFS server, and computer-readable storage medium. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of these embodiments are merely illustrative of the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to the invention without departing from its principles, and these improvements and modifications also fall within the scope of protection of the claims.
Claims
1. A method for writing HDFS data to disk, characterized in that, include: The HDFS server obtains the data to be stored sent by the client device; Determine whether the identifier of the file block to which the data to be stored belongs is in the blocking queue; If so, then the data to be stored is determined to belong to the target file block, and the data to be stored is added to the flush queue and the response queue; the target file block is the file block of the HBase WAL log. If not, then it is determined that the data to be stored does not belong to the target file block, the data to be stored is added to the flush queue, and after the target file block is flushed, the data to be stored is added to the response queue; After adding the data to be stored to the flush queue and the response queue, the process also includes: The data to be stored in the current data to be stored in the data to be stored is processed in segments and the successful data to be stored is checked in a round-robin manner according to the response return information corresponding to the data to be stored in the response queue; wherein, the data to be stored is any of the data to be stored. After all the data to be stored has been flushed, determine whether the successful flush offset corresponding to the data to be stored is equal to the offset corresponding to the data to be stored. If the offset is equal to the offset corresponding to the current data to be stored, then it is determined that the current data to be stored has been successfully flushed, and the storage success information of the current data to be stored is returned to the client device; If the offset is not equal to the offset corresponding to the current data to be stored, it is determined that the current data to be stored has failed to be flushed, and the storage failure information of the current data to be stored is returned to the client device. Also includes: When creating a file, determine whether the file is the HBase WAL log; If it is the HBase WAL log, then the file inode number corresponding to the source code file path of the file is added to the blocking queue; wherein, the identifier is the file block ID.
2. The HDFS data persistence method according to claim 1, characterized in that, The step of determining whether the file is the HBase WAL log includes: Based on the file format, determine whether the file is the HBase WAL log; If it is the HBase WAL log, then the step of adding the file inode number corresponding to the source code file path of the file to the blocking queue is executed.
3. The HDFS data persistence method according to claim 2, characterized in that, The step of determining whether the file is the HBase WAL log based on the file format includes: Determine whether the file format is the preset HBase WAL log format; If the file is the preset HBase WAL log format, then the file is determined to be the HBase WAL log, and the step of adding the file inode number corresponding to the source code file path of the file to the blocking queue is executed.
4. An HDFS data persistence device, characterized in that, Applied to HDFS servers, including: The data acquisition module is used to acquire the data to be stored sent by the client device; The data judgment module is used to determine whether the identifier of the file block to which the data to be stored belongs is in the blocking queue. If yes, it determines that the data to be stored belongs to the target file block; otherwise, it determines that the data to be stored does not belong to the target file block. The target file block is the file block of the HBase WAL log. The log storage module is used to add the data to be stored to the flush queue and the response queue if it belongs to the target file block; A conventional storage module is used to add the data to be stored to the flush queue if it does not belong to the target file block, and to add the data to be stored to the response queue after the target file block is flushed. Also includes: The down-flush polling module is used to perform segmented down-flush of the currently pending data in the down-flush queue, and poll to check the successful down-flush offset corresponding to the currently pending data based on the response return information corresponding to the currently pending data in the response queue; wherein, the currently pending data can be any pending data. The brush-down judgment module is used to determine whether the successful brush-down offset corresponding to the current data to be stored is equal to the offset corresponding to the current data to be stored after all the data to be stored has been brushed down. The successful return module is used to determine that the current data to be stored has been successfully flushed if the value is equal to the offset corresponding to the current data to be stored, and to return the storage success information of the current data to be stored to the client device. The failure return module is used to determine that the current data to be stored has failed to be flushed if the value is not equal to the offset corresponding to the current data to be stored, and to return storage failure information of the current data to be stored to the client device. A recognition module is created to determine whether a file is an HBase WAL log when the file is created. The queue configuration module is used to add the file inode number corresponding to the source code file path of the HBase WAL log to the blocking queue if the log is the HBase WAL log; wherein the identifier is the file block ID.
5. An HDFS server, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the HDFS data persistence method as described in any one of claims 1 to 3 when executing the computer program.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the HDFS data persistence method as described in any one of claims 1 to 3.