Distributed database implementation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a distributed database implementation method and device, electronic equipment and storage medium, relates to the fields of artificial intelligence such as cloud computing, cloud storage and cloud database, and can be applied in an intelligent cloud scene. The method can include: in response to determining that a write operation needs to be performed on a distributed database, writing to-be-written data into a local distributed file system, the local distributed file system and an object storage system being simultaneously mounted on the distributed database; and in response to determining that a primary merge operation needs to be performed on any storage unit in the distributed database, merging files belonging to the storage unit in the local distributed file system and the object storage system, and storing a new file obtained by merging into the object storage system, wherein the storage unit that needs to perform the primary merge operation includes a storage unit that meets a cold storage condition. By applying the scheme, the storage cost can be reduced while the read-write performance is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods, apparatuses, electronic devices and storage media for implementing distributed databases in the fields of cloud computing, cloud storage and cloud databases. Background Technology

[0002] HBase (Hadoop DataBase) is a distributed, column-oriented open-source database. With the development of big data technology, distributed databases have been widely used in the industry. However, their deployment is usually quite complex and storage costs are relatively high. Summary of the Invention

[0003] This disclosure provides methods, apparatus, electronic devices, and storage media for implementing distributed databases.

[0004] A distributed database implementation method includes:

[0005] In response to determining that a write operation needs to be performed on the distributed database, the data to be written is written to the local distributed file system, on which the distributed database is simultaneously mounted the local distributed file system and the object storage system;

[0006] In response to determining that a master merge operation needs to be performed on any storage unit in the distributed database, the files belonging to the storage unit in the local distributed file system and the object storage system are merged, and the merged new file is stored in the object storage system. The storage units that need to perform the master merge operation include storage units that meet the conditions for entering cold storage.

[0007] A distributed database implementation device includes: a first execution module and a second execution module;

[0008] The first execution module is used to write the data to be written to the local distributed file system in response to determining that a write operation needs to be performed on the distributed database. The distributed database is simultaneously mounted with the local distributed file system and the object storage system.

[0009] The second execution module is configured to, in response to determining that a master merge operation needs to be performed on any storage unit in the distributed database, merge the files belonging to the storage unit in the local distributed file system and the object storage system, and store the merged new file in the object storage system. The storage units for which the master merge operation needs to be performed include storage units that meet the conditions for entering cold storage.

[0010] An electronic device, comprising:

[0011] At least one processor; and

[0012] A memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0014] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0015] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0018] Figure 1 This is a flowchart of an embodiment of the HBase implementation method described in this disclosure;

[0019] Figure 2 This is a schematic diagram illustrating the relationship between RegionServer, Store, local HDFS, and object storage system described in this disclosure;

[0020] Figure 3 This is a schematic diagram illustrating the process of performing the MajorCompaction operation and updating the corresponding meta cache as described in this disclosure;

[0021] Figure 4 This is a schematic diagram of the composition structure of the first embodiment 400 of the HBase implementation device described in this disclosure;

[0022] Figure 5 This is a schematic diagram of the composition structure of the second embodiment 500 of the HBase implementation device described in this disclosure;

[0023] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0026] Figure 1 This is a flowchart illustrating an embodiment of the HBase implementation method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0027] In step 101, in response to determining that a write operation needs to be performed on HBase, the data to be written is written to the local distributed file system (HDFS, Hadoop Distributed File System). HBase is simultaneously mounted with the local HDFS and the object storage system (Object Storage).

[0028] In step 102, in response to determining that a major compaction operation needs to be performed on any storage unit (Store) in HBase, the files (HFiles) belonging to that Store in the local HDFS and object storage system are merged, and the merged new HFiles are stored in the object storage system. The Stores that need to perform the major compaction operation include those that meet the conditions for entering cold storage.

[0029] By adopting the scheme described in the above method embodiment, relatively hot data can be stored in local HDFS, taking advantage of the fast read and write speed of local HDFS to ensure read and write performance, while cold data can be stored in the object storage system, thereby reducing storage costs. In other words, it takes into account the advantages of both local HDFS and object storage system, thereby improving the performance of HBase.

[0030] Object storage, also known as object-based storage, is a general term used to describe methods for solving and processing discrete units, which can be called objects. Object storage can support various types of data storage, such as text, multimedia, and binary data, and is suitable for various scenarios such as data distribution, data backup, big data analysis, and audio and video file processing. Object storage usually has the function of hierarchical storage, that is, cold data can be gradually moved to more cost-effective storage media according to the access frequency of data, so as to reduce storage costs.

[0031] Local HDFS is the original underlying storage method of HBase. In the solution described in this disclosure, an object storage system is further added, that is, both local HDFS and object storage system are mounted on HBase.

[0032] Preferably, in response to determining that the following operations for HBase need to be performed: operations that affect the file tree structure, the operations can be performed using a double-write approach.

[0033] Operations that affect the file tree structure may include creating tables, deleting tables, splitting regions, and migrating regions.

[0034] By employing a dual-write approach, i.e. performing dual-write operations, the file tree structure of the local HDFS and the object storage system can be ensured to be consistent. Since the operations affecting the file tree structure account for a very small proportion of the total number of operations, the latency caused by dual-write operations is almost negligible.

[0035] When a write operation needs to be performed on HBase, the data to be written can be written to the local HDFS. For example, it can be written to the Write-Ahead Log (WAL) and the write cache (memstore) in the original path in the traditional way, and then flushed into the local HDFS.

[0036] When it is determined that a MajorCompaction operation needs to be performed on any Store in HBase, HFiles belonging to that Store in the local HDFS and object storage system can be merged, and the merged new HFile can be stored in the object storage system.

[0037] For HBase, the best read efficiency is usually achieved when each Store contains only one HFile. Therefore, HBase reduces the disk seek time for each read by merging existing HFiles, thereby improving read speed. This merging process is called compaction.

[0038] HBase compaction operations can be divided into two types: minor compaction operations and major compaction operations. The timing and specific type of compaction operation can be determined based on a pre-defined strategy.

[0039] When a MinorCompaction operation is required, it can be performed in the traditional way, merging only HFiles on the local HDFS, such as automatically checking small files and merging them into large files, thereby reducing fragmented files.

[0040] When a MajorCompaction operation is performed, HFiles belonging to the same Store in both the local HDFS and object storage systems can be merged, and the resulting new HFile can be stored in the object storage system. In other words, the MajorCompaction operation retrieves all HFiles from the same Store in both the local HDFS and object storage systems, merges them to obtain a new HFile, and then writes the new HFile to the object storage system. Correspondingly, the old HFiles from before the merge are no longer retained.

[0041] The compaction operation can be performed by a dedicated compaction thread, without affecting data reading, meaning users can still obtain data from the old HFile.

[0042] Preferably, each Store in HBase can be monitored in real time. In response to determining that any monitored Store meets the conditions for entering cold storage, the Store can be identified as the Store that needs to perform Major Compaction operation.

[0043] In practical applications, a monitoring thread can be started to monitor the status of each Store in real time and determine whether each Store meets the conditions for entering cold storage. If any Store meets the conditions for entering cold storage, a MajorCompaction operation can be triggered for that Store, thereby sending the Store's HFile to the object storage system. As mentioned earlier, the MajorCompaction operation can be handled by the compaction thread without affecting the monitoring thread.

[0044] The monitoring thread can be created when the RegionServer starts. The specific conditions for entering cold storage can be determined based on actual needs and can be predefined. For example, for any Store, if the difference between the last read time and the current time is greater than a predetermined threshold, then the Store can be determined to meet the conditions for entering cold storage. Similarly, if the local HDFS space is about to run out, i.e., the remaining space is less than a predetermined threshold, it can also be considered to meet the conditions for entering cold storage. The specific value of the threshold can also be determined based on actual needs.

[0045] The above processing can separate hot and cold data, thereby reducing storage costs.

[0046] Preferably, for any Store, when performing MajorCompaction on HFiles belonging to that Store in the local HDFS and the object storage system, the files belonging to that Store in the local HDFS can be determined based on the file tree structure in the local HDFS, and the files belonging to that Store in the object storage system can be determined based on the file tree structure in the object storage system. Since the file tree structures of the local HDFS system and the object storage system are consistent, the HFiles belonging to that Store can be merged accordingly.

[0047] The above processing ensures the accuracy of the obtained HFile, thereby improving the accuracy of the Major Compaction operation results.

[0048] In addition, preferably, for any Store, when performing MajorCompaction operation on HFiles belonging to that Store in the local HDFS and the object storage system, the HFiles belonging to that Store in the local HDFS can be obtained first, and the obtained HFiles can be merged for the first time. During the first merge, the HFiles belonging to that Store in the object storage system can be obtained. Then, the result of the first merge can be merged with the obtained HFiles belonging to that Store in the object storage system for the second time, and the result of the second merge can be stored as a new HFile in the object storage system.

[0049] Generally speaking, object storage systems have relatively high latency. Therefore, it is advisable to first merge the HFiles in the local HDFS, i.e., perform the first merge, and then perform the second merge with the HFiles in the object storage system. In this process, the time of the first merge can be used to retrieve the HFiles from the object storage system, thereby saving the time of waiting for the object storage system to read the files and thus improving the merging efficiency.

[0050] Preferably, in response to determining that a read operation for HBase needs to be performed, the view objects can be traversed in the following order: cache, HFile in local HDFS, and object storage system. The cache includes HBase's write cache and read cache. For each view object that has been traversed, in response to determining that a read result corresponding to the read operation exists therein, the read result can be obtained from the view object and returned.

[0051] When it is determined that a read operation for HBase needs to be performed, the cache can be checked first to determine whether a hit occurs. The cache may include HBase's write cache and read cache. For example, the write cache may be memstore, and the read cache may be block cache and bucket cache. If a hit occurs, the read result can be obtained from it and returned.

[0052] If the cache misses, check if the HFile in the local HDFS is also a match. If it is, retrieve the read result from it and return it.

[0053] If the file is not found in the local HDFS HFile, you can further check if it is found in the object storage system. If it is found, you can retrieve the read result from it and return it.

[0054] This allows you to view objects in descending order of reading efficiency, prioritizing the use of objects with higher reading efficiency to obtain reading results, thereby improving the efficiency of obtaining reading results and thus improving the response speed of read operations.

[0055] If no match is found after checking each object in turn, the system can return messages such as "No read result found" or "Read failed" to notify the user.

[0056] Preferably, when viewing an object storage system, the existence of a read result in the object storage system can be determined by querying the metadata cache (metacache) in the local HDFS. In response to the determination that the result exists, the read result can be retrieved from the object storage system and returned. Each Store can correspond to a metacache, which is used to store the directory information of HFiles belonging to that Store in the object storage system.

[0057] The directory information may include index blocks and bloom blocks. An HFile consists of multiple blocks, such as index blocks, bloom blocks, and data blocks, with the actual data stored in the data blocks.

[0058] Because object storage systems have high latency, if directory information is stored in an object storage system, it may be necessary to frequently read HFiles from the object storage system, which will greatly reduce performance. Therefore, in the solution described in this disclosure, a meta cache can be created in the local HDFS for each Store to store metadata such as index blocks and bloom blocks of HFiles belonging to that Store in the object storage system. Considering that the amount of metadata is usually small, creating a meta cache in the local HDFS will not occupy too much extra space. Moreover, if the block cache has sufficient space, it is also possible to keep the metadata of HFiles in the object storage system permanently in the block cache.

[0059] Preferably, in response to determining that a MajorCompaction operation has been performed for any Store, the metacache corresponding to that Store can be regenerated, and / or, in response to determining that the RegionServer has restarted, the metacache corresponding to each Store can be regenerated.

[0060] When performing a compaction operation, the information in the metacache also needs to be updated synchronously to improve the accuracy of the information in the metacache. In practical applications, transactions within a region can be used to write the new HFile to the object storage system. Afterwards, the original metacache can be deleted and a new metacache can be regenerated, which is to update the metacache. In addition, to prevent the HFile in the object storage system from being inconsistent with the metacache due to a crash halfway through the transaction, the metacache of each store can be regenerated based on the directory information obtained from the object storage system every time the RegionServer is restarted. This operation can be asynchronous to speed up the restart speed of the RegionServer.

[0061] Preferably, when it is determined that a read operation for HBase needs to be performed, in response to the determination that metacache does not exist in the local HDFS, metacache can be generated first, and then the generated metacache can be queried to determine whether the read result exists in the object storage system.

[0062] The system can first obtain the required directory information from the object storage system, then generate the corresponding metacache based on the obtained directory information and store it in the local HDFS. Then, by querying the generated metacache, it can determine whether the required read result exists in the object storage system, thereby improving the success rate of returning the read result to the user.

[0063] The above situation usually only occurs when the RegionServer has just finished restarting or when the Major Compaction operation has just ended. Accordingly, the required meta cache can be generated using the corresponding thread.

[0064] Based on the above introduction, Figure 2 This is a schematic diagram illustrating the relationship between RegionServer, Store, local HDFS, and object storage system described in this disclosure.

[0065] like Figure 2 As shown in the attached diagram, each RegionServer can correspond to one or more Stores. (This is for simplification.) Figure 2 The code only represents one Store. Each Store can include one memstore and one or more storage files (StoreFile). Each StoreFile corresponds to an HFile in the local HDFS. In other words, StoreFile and HFile are essentially the same, but they are called StoreFile in the Store and HFile in the local HDFS. In addition, each StoreFile can be used as an object in the object storage system. Furthermore, metacache can be added to the local HDFS.

[0066] Each RegionServer can correspond to one blockcache and one bucketcache, which are not shown in the accompanying diagram for simplicity.

[0067] Figure 3 This is a schematic diagram illustrating the process of performing the MajorCompaction operation and updating the corresponding metacache as described in this disclosure. Figure 3 As shown, 1 represents the first merge, 2 represents the second merge, 3 represents writing the result of the second merge to the object storage system, 4 represents deleting the original meta cache, and 5 represents regenerating the new meta cache. It can be seen that due to the MajorCompaction operation, for each Store, there is almost always only one HFile in the object storage system.

[0068] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0069] In summary, the solution described in the embodiments of this disclosure can reduce storage costs while ensuring HBase read and write performance as much as possible, thereby improving the performance and cost-effectiveness of HBase. In addition, the cold storage uses HBase's file tree structure, which can easily convert the object storage system into an offline data warehouse, thereby providing a data source for big data analysis.

[0070] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0071] Figure 4 This is a schematic diagram of the structural composition of the first embodiment 400 of the HBase implementation device described in this disclosure. Figure 4 As shown, it includes: a first execution module 401 and a second execution module 402.

[0072] The first execution module 401 is used to respond to the determination that a write operation for HBase needs to be performed by writing the data to be written to the local HDFS. HBase is simultaneously mounted with the local HDFS and the object storage system.

[0073] The second execution module 402 is used to merge the HFiles belonging to the Store in the local HDFS and the object storage system in response to the determination that a MajorCompaction operation needs to be performed on any Store in HBase, and to store the merged new HFile in the object storage system. The Stores that need to perform the MajorCompaction operation include Stores that meet the conditions for entering cold storage.

[0074] By adopting the scheme described in the above device embodiment, relatively hot data can be stored in local HDFS, taking advantage of the fast read and write speed of local HDFS to ensure read and write performance, while cold data can be stored in the object storage system, thereby reducing storage costs. In other words, it takes into account the advantages of both local HDFS and object storage system, thereby improving the performance of HBase.

[0075] When a write operation needs to be performed on HBase, the first execution module 401 can write the data to be written to the local HDFS, which can be done in the traditional way.

[0076] When it is determined that a MajorCompaction operation needs to be performed on any Store in HBase, the second execution module 402 can merge the HFiles belonging to that Store in the local HDFS and the object storage system, and can store the merged new HFiles in the object storage system.

[0077] HBase compaction operations can be divided into two types: MinorCompaction and MajorCompaction. The timing and specific type of compaction operation can be determined based on a pre-defined strategy.

[0078] When the MinorCompaction operation needs to be performed, the second execution module 402 can merge only the HFiles on the local HDFS in the traditional way, such as automatically checking small files and merging them into large files, thereby reducing fragmented files.

[0079] When a MajorCompaction operation is required, the second execution module 402 can merge HFiles belonging to the same Store in the local HDFS and object storage system, and store the merged new HFile in the object storage system. In other words, the MajorCompaction operation retrieves all HFiles under the same Store in the local HDFS and object storage system, merges them to obtain a new HFile, and then writes the new HFile to the object storage system. Correspondingly, the old HFiles from before the merge will no longer be retained.

[0080] Preferably, the second execution module 402 can monitor each Store in HBase in real time. In response to determining that any of the monitored Stores meets the conditions for entering cold storage, the Store can be identified as the Store for which MajorCompaction operation needs to be performed.

[0081] In practical applications, a monitoring thread can be started to monitor the status of each store in real time and determine whether each store meets the conditions for entering cold storage. If any store meets the conditions for entering cold storage, a MajorCompaction operation can be triggered for that store.

[0082] Preferably, for any Store, when the second execution module 402 performs MajorCompaction on HFiles belonging to that Store in the local HDFS and the object storage system, it can determine the files belonging to that Store in the local HDFS based on the file tree structure in the local HDFS, and determine the files belonging to that Store in the object storage system based on the file tree structure in the object storage system. Since the file tree structures of the local HDFS system and the object storage system are consistent, the determined HFiles belonging to that Store can be merged accordingly.

[0083] Preferably, in response to determining that the following operations for HBase need to be performed: operations that affect the file tree structure, the second execution module 402 may use a double-write method to complete the operations.

[0084] In addition, preferably, for any Store, when the second execution module 402 performs MajorCompaction operation on the HFiles belonging to the Store in the local HDFS and the object storage system, it can first obtain the HFiles belonging to the Store in the local HDFS, and can perform a first merge on the obtained HFiles. During the first merge, the HFiles belonging to the Store in the object storage system can be obtained. Then, the result of the first merge can be merged with the obtained HFiles belonging to the Store in the object storage system for a second time, and the result of the second merge can be stored as a new HFile in the object storage system.

[0085] Figure 5 This is a schematic diagram of the structural composition of the second embodiment 500 of the HBase implementation device described in this disclosure. Figure 5 The aforementioned includes: a first execution module 401, a second execution module 402, and a third execution module 403.

[0086] Among them, the first execution module 401 and the second execution module 402 and Figure 4 The same applies to the embodiments shown.

[0087] The third execution module 403 is used to respond to the determination that a read operation for HBase needs to be performed, and can traverse each view object in the following order: cache, HFile in local HDFS, and object storage system; the cache includes HBase's write cache and read cache. For each view object that is traversed, in response to the determination that there is a read result corresponding to the read operation, the read result can be obtained from the view object and returned.

[0088] When it is determined that a read operation for HBase needs to be performed, the cache can be checked first to determine whether a hit occurs. The cache may include HBase's write cache and read cache. If a hit occurs, the read result can be obtained from it and returned.

[0089] If the cache misses, check if the HFile in the local HDFS is also a match. If it is, retrieve the read result from it and return it.

[0090] If the file is not found in the local HDFS HFile, you can further check if it is found in the object storage system. If it is found, you can retrieve the read result from it and return it.

[0091] Preferably, when the object being viewed is an object storage system, the third execution module 403 can determine whether a read result exists in the object storage system by querying the metacache in the local HDFS. In response to the determination that the result exists, the read result can be retrieved from the object storage system and returned. Each Store can correspond to a metacache, which is used to store the directory information of HFiles belonging to that Store in the object storage system. The directory information may include indexblock and bloomblock.

[0092] Preferably, the third execution module 403 may regenerate the metacache corresponding to any Store in response to determining that a Major Compaction operation has been completed for any Store, and / or may regenerate the metacache corresponding to each Store in response to determining that the RegionServer has restarted.

[0093] Preferably, the third execution module 403 determines that a read operation for HBase needs to be performed. In response to determining that metacache does not exist in the local HDFS, it can first generate metacache, and then query the generated metacache to determine whether the read result exists in the object storage system.

[0094] Figure 4 and Figure 5 The specific workflow of the device embodiment shown can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0095] In summary, by adopting the solution described in the embodiments of this disclosure, storage costs can be reduced while ensuring HBase read and write performance as much as possible, thereby improving the performance and cost-effectiveness of HBase. In addition, the cold storage adopts the file tree structure of HBase, which can easily convert the object storage system into an offline data warehouse, thereby providing a data source for big data analysis, etc.

[0096] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly in areas such as cloud computing, cloud storage, and cloud databases. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0097] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0098] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0099] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0100] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0101] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the methods described in this disclosure by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0108] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for implementing a distributed database, comprising: In response to determining that a write operation needs to be performed on the distributed database, the data to be written is written to the local distributed file system, on which the distributed database is simultaneously mounted the local distributed file system and the object storage system; In response to determining that a minor merge operation is required, files on the local distributed file system are merged. In response to determining that a major merge operation is required for any storage unit in the distributed database, files belonging to the storage unit in the local distributed file system and the object storage system are merged according to the file tree structure in the local distributed file system and the file tree structure in the object storage system, and the merged new file is stored in the object storage system. The storage units for which the major merge operation is required include storage units that meet the conditions for entering cold storage. It also includes: in response to determining that the following operations for the distributed database need to be performed: operations affecting the file tree structure, the operations are performed using a dual-write approach using the local distributed file system and the object storage system; It also includes: in response to determining that a read operation needs to be performed on the distributed database, sequentially traversing the following viewing objects: cache, files in the local distributed file system, and the object storage system; wherein, when the viewing object is the object storage system, by querying the metadata cache in the local distributed file system, it is determined whether a read result exists in the object storage system; in response to determining that the result exists, the read result is obtained from the object storage system and returned; each storage unit corresponds to one metadata cache, which is used to store the directory information of the files belonging to the storage unit in the object storage system.

2. The method according to claim 1, wherein, The merging of files belonging to the storage unit in the local distributed file system and the object storage system includes: Based on the file tree structure in the local distributed file system, the files belonging to the storage unit in the local distributed file system are determined, and based on the file tree structure in the object storage system, the files belonging to the storage unit in the object storage system are determined, wherein the file tree structure of the local distributed file system is consistent with that of the object storage system. The files that are identified as belonging to the storage unit are merged.

3. The method according to claim 1, wherein, The determination that a master merge operation needs to be performed on any storage unit in the distributed database includes: Each storage unit in the distributed database is monitored in real time. In response to determining that any monitored storage unit meets the conditions for entering cold storage, the storage unit is identified as the storage unit for which the master merge operation needs to be performed.

4. The method according to claim 1, wherein, The step of merging the files belonging to the storage unit in the local distributed file system and the object storage system, and storing the merged new file in the object storage system includes: The files belonging to the storage unit in the local distributed file system are obtained, and the obtained files are merged for the first time. During the first merge, the files belonging to the storage unit in the object storage system are also obtained. The first merge result is merged with the file belonging to the storage unit in the object storage system, and the second merge result is stored as the new file in the object storage system.

5. The method according to any one of claims 1 to 4, wherein, The cache includes the write cache and read cache of the distributed database.

6. The method according to claim 1, further comprising: In response to determining that the primary merge operation has been completed for any storage unit, the metadata cache corresponding to the storage unit is regenerated; And / or, in response to a determined region server restart, regenerate the metadata cache corresponding to each storage unit.

7. The method according to claim 1, further comprising: In response to determining that the metadata cache does not exist in the local distributed file system, the metadata cache is generated, and the existence of the read result in the object storage system is determined by querying the generated metadata cache.

8. A distributed database implementation apparatus, comprising: The first execution module, the second execution module, and the third execution module; The first execution module is used to write the data to be written to the local distributed file system in response to determining that a write operation needs to be performed on the distributed database. The distributed database is simultaneously mounted with the local distributed file system and the object storage system. The second execution module is configured to merge files on the local distributed file system in response to determining that a smaller merge operation needs to be performed, and to merge files belonging to the storage unit in the local distributed file system and the object storage system according to the file tree structure in the local distributed file system and the file tree structure in the object storage system, and to store the merged new file in the object storage system. The storage units for which the main merge operation needs to be performed include storage units that meet the conditions for entering cold storage. The second execution module is further configured to, in response to determining that the following operations need to be performed on the distributed database: operations affecting the file tree structure, complete the operations using a dual-write method of the local distributed file system and the object storage system; The third execution module is configured to, in response to determining that a read operation needs to be performed on the distributed database, sequentially traverse the following viewing objects: cache, files in the local distributed file system, and the object storage system; wherein, when the viewing object is the object storage system, it determines whether a read result exists in the object storage system by querying the metadata cache in the local distributed file system, and in response to determining that the result exists, it retrieves the read result from the object storage system and returns it, with each storage unit corresponding to a metadata cache for storing directory information of files belonging to the storage unit in the object storage system.

9. The apparatus according to claim 8, wherein, The second execution module determines the files belonging to the storage unit in the local distributed file system based on the file tree structure in the local distributed file system, and determines the files belonging to the storage unit in the object storage system based on the file tree structure in the object storage system. The file tree structure of the local distributed file system is consistent with that of the object storage system. The determined files belonging to the storage unit are then merged.

10. The apparatus according to claim 8, wherein, The second execution module monitors each storage unit in the distributed database in real time. In response to determining that any monitored storage unit meets the conditions for entering cold storage, the storage unit is identified as the storage unit for which the master merge operation needs to be performed.

11. The apparatus according to claim 8, wherein, The second execution module obtains the files belonging to the storage unit in the local distributed file system and performs a first merge on the obtained files. Simultaneously with the first merge, it obtains the files belonging to the storage unit in the object storage system, performs a second merge with the obtained files belonging to the storage unit in the object storage system, and stores the second merge result as the new file in the object storage system.

12. The apparatus according to any one of claims 8 to 11, wherein, The cache includes the write cache and read cache of the distributed database.

13. The apparatus according to claim 8, wherein, The third execution module is further configured to, in response to determining that the main merge operation has been completed for any storage unit, regenerate the metadata cache corresponding to the storage unit, and / or, in response to determining that the region server has restarted, regenerate the metadata cache corresponding to each storage unit.

14. The apparatus according to claim 8, wherein, The third execution module is further configured to, in response to determining that the metadata cache does not exist in the local distributed file system, generate the metadata cache, and determine whether the read result exists in the object storage system by querying the generated metadata cache.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-7.

Images