Metadata access methods, devices, electronic equipment and storage media

Abstract

This application relates to a metadata access method, apparatus, electronic device, and storage medium. The method includes: obtaining a data access request forwarded by a preset Waggle Dance to a target MetaStore instance, the data access request carrying metadata target information of the target metadata requested by the client; parsing the metadata target information from the data access request, and reconstructing a query request for querying the target database table shard storing the target metadata based on the metadata target information; and querying the target database table shard among multiple database table shards storing metadata according to the query request, so as to obtain the target metadata after accessing the target database table shard, wherein the multiple database table shards are generated by partitioning the metadata database corresponding to the target MetaStore instance according to a preset sharding rule. This application solves the problems of slow metadata access speed and poor Hive stability when the metadata database data storage volume is large.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a metadata access method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, Hive is a data warehouse tool based on Hadoop (a distributed system infrastructure developed by the Apache Software Foundation). It can map structured data files to a database table and provide Structured Query Language (SQL) query functionality. It can transform SQL statements into MapReduce tasks, a computing model for parallel big data processing, for execution. Hive's metadata is stored in a relational database (e.g., MySQL) and accessed through Hive's MetaStore (a service used by Hive to manage database and table metadata). The Hive MetaStore service mainly consists of multiple HiveMetaStore instances connected by load balancing technology. All HiveMetaStore instances share a master-slave MySQL environment as the Hive metadata storage.

[0003] In related technologies, as related data services continue to grow, the amount of data in the Hive metadata database is increasing, sometimes resulting in a single table storing more than 100 million records. This can lead to a significant drop in MySQL performance, causing slow query or access to metadata. At the same time, during peak business periods, the CPU utilization of the machine often reaches 100%, seriously affecting service stability.

[0004] There is still no effective solution to the problems of high query pressure on MySQL, slow metadata access speed, and poor Hive stability when the Hive metadata database has a large amount of data storage. Summary of the Invention

[0005] This application provides a metadata access method, apparatus, electronic device, and storage medium to at least solve the problems in related technologies such as high query pressure on MySQL, slow metadata access speed, and poor Hive stability when the data storage volume of the Hive metadata database is large.

[0006] In a first aspect, this application provides a metadata access method, comprising: obtaining a data access request forwarded by a preset federated routing management server Waggle Dance to a target metadata storage MetaStore instance, wherein the data access request carries metadata target information of the target metadata requested by the client; parsing the metadata target information from the data access request, and reconstructing a query request for querying a target database table shard storing the target metadata based on the metadata target information; and querying the target database table shard among multiple database table shards storing metadata according to the query request, so as to obtain the target metadata after accessing the target database table shard, wherein the multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to a preset sharding rule.

[0007] Secondly, this application provides a metadata access device, comprising:

[0008] The acquisition module is used to acquire data access requests forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, wherein the data access request carries the meta target information of the target metadata requested by the client;

[0009] The processing module is used to parse the metadata target information from the data access request and reconstruct the query request for querying the target database table shard that stores the target metadata based on the metadata target information.

[0010] The access module is used to query the target database table shard from multiple database table shards storing metadata according to the query request, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to a preset sharding rule.

[0011] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0012] Memory, used to store computer programs;

[0013] When a processor executes a program stored in memory, it implements the steps of the metadata access method described in any embodiment of the first aspect.

[0014] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the metadata access method as described in any embodiment of the first aspect.

[0015] Compared with related technologies, the metadata access method, apparatus, electronic device, and storage medium provided in this embodiment solve the problems of high query pressure, slow metadata access speed, and poor Hive stability when the Hive metadata database has a large data storage volume. This is achieved by acquiring a data access request forwarded by a preset federated routing management server (Waggle Dance) to a target metadata storage (MetaStore) instance, wherein the data access request carries metadata target information requested by the client. The metadata target information is parsed from the data access request, and a query request for querying the target database table shard storing the target metadata is reconstructed based on the metadata target information. Based on the query request, the target database table shard is queried from multiple database table shards storing metadata, so that the target metadata can be obtained after accessing the target database table shard. The multiple database table shards are generated by partitioning the metadata database corresponding to the target MetaStore instance according to preset sharding rules. This solves the problems of high MySQL query pressure, slow metadata access speed, and poor Hive stability when the Hive metadata database has a large data storage volume, achieving the beneficial effects of decompressing the metadata database, improving Hive stability, and increasing the speed of metadata query and access.

[0016] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the metadata access method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the interaction between Waggle_dance and Hive MetaStore in an embodiment of this application. Figure 1 ;

[0021] Figure 3 This is a schematic diagram of the interaction between Waggle_dance and Hive MetaStore in an embodiment of this application. Figure 2 ;

[0022] Figure 4 This is a schematic diagram of the library segmentation in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the metadata access process in a preferred embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the intermediate controller for database sharding and table partitioning in an embodiment of this application;

[0025] Figure 7 This is a structural block diagram of a metadata access device according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Before describing the embodiments of this application, the following explains the terminology involved in this application and the implementation process of existing metadata access schemes:

[0029] Hive: Hive is a data warehouse tool based on Hadoop. It maps structured data files to a database table and provides SQL query functionality, transforming SQL statements into MapReduce tasks for execution. Hive's metadata is stored in a relational database (e.g., MySQL) and accessed through the Hive MetaStore. The Hive MetaStore service mainly consists of multiple Hive MetaStore instances connected by load balancing technology. All Hive MetaStore instances share a master-slave MySQL environment as the Hive metadata storage.

[0030] Hive MetaStore: Hive Metastore is a service used by Hive to manage database and table metadata, which is generally stored using a relational database.

[0031] HiveServer2: A server-side interface that allows remote clients to execute queries on Hive and return results.

[0032] Hadoop: A distributed system infrastructure developed by the Apache Software Foundation. Users can develop distributed programs without understanding the underlying details of distributed systems, and make full use of the power of clusters for high-speed computing and storage. Hadoop provides the distributed storage framework HDFS, the distributed computing framework MapReduce, and the distributed scheduling framework Yarn.

[0033] Waggle_dance is a routing service for Hive MetaStore. It fully implements the Thrift API of HiveMetaStore, and users can directly access Waggle Dance when they need to access Hive MetaStore.

[0034] SQL: Structured Query Language is a special-purpose programming language, a database query and programming language used to access, query, update, and manage relational database systems.

[0035] The existing metadata access process is as follows:

[0036] The first step is for the client to write SQL (data) and submit it;

[0037] The second step is for the Hiveserver2 port to receive the SQL submitted by the client, communicate with the HiveMetaStore service, and expect to obtain metadata.

[0038] The third step involves the HiveMetaStore service receiving requests from HiveServer2, converting them into corresponding SQL queries, querying the metadata database, and returning the results.

[0039] The fourth step is that after Hiveserver2 obtains the metadata, it compiles and parses the SQL, converting it into a physical execution plan;

[0040] Fifth, HiveServer2 submits the physical execution plan to the distributed query engine for execution;

[0041] The sixth step is that after the query engine finishes executing, it returns the results to HiveServer2, which then processes the results and returns them to the client.

[0042] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart illustrating the metadata access method according to an embodiment of this application. Figure 1As shown in the figure, this application provides a metadata access method, which includes the following steps:

[0044] Step S101: Obtain the data access request forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, wherein the data access request carries the meta target information of the target metadata requested by the client.

[0045] In this embodiment, the execution entity for the metadata access method of this application is a sharded intermediate controller added between multiple MetaStore instances and the corresponding metadata database. In this embodiment, before the controller executes the access method process of this application, the Waggle Dance corresponding to Hive forwards the data access request to the corresponding MetaStore instance according to the request type corresponding to the data access request submitted by the client. The request type refers to the permission of the configured MetaStore instance to read and write metadata in the associated metadata database, such as read-only or read / write. In some optional methods, Waggle Dance can also determine the MetaStore instance to which the data access request needs to be forwarded based on the data name of the target metadata metadata requested by the client in the data access request, that is, route the data access request initiated by the client to the corresponding MetaStore instance.

[0046] In this embodiment, the sharding and table partitioning intermediate controller intercepts the data access request when the target MetaStore instance is expected to send the data access request to its associated metadata database to query user data, thereby obtaining the data access request.

[0047] In this embodiment, the existing Hive metadata database (Hive DB) is partitioned to distribute metadata across multiple MySQL environments at the Hive DB level. Each MySQL environment includes a MetaStore instance and a corresponding MySQL instance. Furthermore, this embodiment configures multiple master Metastore instances and multiple slave Metastore instances. The MySQL instance corresponding to the master Metastore instance is the master MySQL instance, and the master Metastore instance performs read-write operations on the master MySQL instance. The MySQL instances corresponding to the slave Metastore instances are slave MySQL instances, and the slave Metastore instances perform read-only operations on the slave MySQL instance. In this embodiment, by separating master and slave Metastore instances, the access pressure on the metadata database is reduced, and by setting up multiple master and slave Metastore instances, service stability is ensured.

[0048] It should be noted that in this embodiment, Waggle Dance is equivalent to a routing service of Hive Metastore, which implements the Metastore's Thrift API. When users need to access the Metastore, they directly access Waggle Dance. Waggle Dance is connected to multiple Hive Metastores, combining these originally unconnected Metastores into a whole. Thrift uses an intermediate language (Interface Definition Language IDL) to define the RPC interface and data types, and then a compiler generates code in different languages, which is responsible for the implementation of the RPC protocol layer and transport layer. API refers to the application programming interface.

[0049] Step S102: Parse the metadata target information from the data access request, and reconstruct the query request for querying the target database table shards that store the target metadata based on the metadata target information.

[0050] In this embodiment, after a data access request is obtained, the data access request is parsed to obtain the metadata target information corresponding to the target metadata to be accessed. That is, by parsing the data access request, at least the data name of the metadata stored in the metadata database to be accessed can be known. In some embodiments, the metadata and its metadata target information include, but are not limited to: the name of the database table, the columns and partitions of the database table and their attributes, the attributes of the database table (internal table and external table), and the directory where the data of the database table is located.

[0051] In this embodiment, after parsing the target metadata, it is necessary to determine the target database table shard where the target metadata is stored, that is, to determine the physical storage address for accessing the target metadata (e.g., a target database table shard horizontally partitioned from the corresponding metadata database). Then, the sharding intermediate controller generates a corresponding query request based on the physical storage address, communicates with the corresponding metadata database based on the query request, and queries the metadata database to obtain the corresponding target metadata. In this embodiment, the corresponding metadata database is a vertically partitioned metadata database of Hive, distributed across different nodes (also called a sharding database). At the same time, the metadata database serves as the storage for the corresponding Metastore instance, meaning that multiple Metastore instances of Hive do not share a single master-slave MsSQL database, but rather each Metastore instance corresponds to a MsSQL database. By performing sharding at the Hive DB level, the access pressure on the metadata database is reduced, that is, different source databases are accessed according to different needs.

[0052] In this embodiment, vertically partitioning the Hive DB (generating multiple database shards) cannot solve the problem of slow metadata querying or access caused by the excessive data volume of a single database table in the corresponding metadata database. Therefore, in this embodiment, the database tables in each vertically partitioned metadata database are also horizontally split, thereby generating multiple database table shards for storing metadata. Each database table shard corresponds to one table. For example, taking the TBLS table as an example, before horizontal splitting, the corresponding single database table is DB0. After horizontal splitting into 3 tables, the corresponding database table shards are: TAB_0, TAB_1, and TAB_2. In this embodiment, after completing... After horizontal partitioning of the database tables, one MetaStore instance corresponds to multiple database table shards. In some optional embodiments, when generating database table shards through horizontal partitioning, taking TBLS tables as an example, a hash partitioning method is used to generate database table shards. Specifically, the table code ID of a database table (representing metadata in the form of a table) corresponding to a certain metadata is hashed and then moduloed with the number of table shards (e.g., modulo operation) to distribute the metadata evenly across different database table shards. For example, if the table code ID (denoted as idtable_id) of a database table is 10, and the corresponding horizontal partitioning generates database table shards: TAB_0, TAB_1, and TAB_2, and the corresponding number of table shards shared_num is 3, then 10 mod 3 = 1, which means that the data of this database table should be stored in the partitioned database table shard TAB_1. It can be understood that, from the perspective of metadata storage, using a database partitioning and table partitioning approach avoids the slow query speed and service instability caused by a single MySQL database with massive amounts of data.

[0053] Of course, it is understandable that sharding and partitioning Hive DB is done after the Hive DB architecture and corresponding metadata storage have been completed. In this embodiment, the corresponding metadata access is completed by performing a lookup of the target database table shards.

[0054] Step S103: Based on the query request, the target database table shard is queried from multiple database table shards storing metadata, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to preset sharding rules.

[0055] In this embodiment, after parsing the data access request, the sharding intermediate controller determines the target database shard where the target metadata is located. It then accesses the metadata database where the target database shard is located, which is the metadata database corresponding to the target MetaStore instance, and then accesses the target database shard to obtain the target metadata.

[0056] In this embodiment, the preset sharding rules are at least the generation rules corresponding to the shards in the above-mentioned database sharding and table splitting process. In this embodiment, the execution of query or access operations is to determine the corresponding target database table shard based on the metadata of the target to be accessed. Correspondingly, the database sharding and table splitting intermediate controller converts the logical table name corresponding to the target database table shard into the physical table name on the actual storage according to the preset sharding rules and data access SQL. Finally, the query SQL is rewritten and executed via JDBC to obtain the target metadata or access results.

[0057] Through steps S101 to S103, a data access request forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance is obtained. The data access request carries the target metadata information requested by the client. The target metadata information is parsed from the data access request, and the query request for querying the target database table shard of the target metadata is reconstructed based on the target metadata information. According to the query request, the target database table shard is queried from multiple database table shards storing metadata, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to preset sharding rules. This solves the problems of high query pressure, slow metadata access speed, and poor Hive stability when the Hive metadata database data storage volume is large. It achieves the beneficial effects of decompressing the metadata database, improving Hive stability, and improving the speed of metadata query and access.

[0058] It should be noted that in this embodiment, a federated approach is adopted at the metadata access level to achieve horizontal scaling of metadata, decompress the MySQL database, and improve the stability of Hive. The Hive DB is split, and the metadata is divided into multiple MySQL environments for storage at the Hive DB level. Without changing the original Hive Metastore interface (Hiveserver2), client requests are routed to multiple Hive metastore storage environments and read and write are supported. At the metadata storage level, a database sharding and table partitioning approach is adopted to avoid the slow query speed and service instability caused by a single MySQL database for massive data.

[0059] In some embodiments, the query request for querying the target database table shard that stores the target metadata is reconstructed based on the metadata target information, which is achieved through the following steps:

[0060] Step 21: Detect the database table name of the target database table shard corresponding to the target metadata in the target metadata.

[0061] Step 22: Generate the physical storage information corresponding to the target database table shard based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards. The physical storage information is used to characterize the storage location of the target database table shard in the metadata database corresponding to the target MetaStore instance.

[0062] In this embodiment, the logical table number corresponds to a logical table that does not exist in the physical storage space. This logical table number is determined based on the mapping relationship between multiple database table shards and the metadata stored in each database table shard, which is the position or arrangement position of the target database table shard in the overall mapping relationship table after database sharding. In this embodiment, it is necessary to convert the table name of the logical table corresponding to the logical table number into the physical table name of the actual stored physical table, that is, to determine the actual physical storage location of the target database table shard. In this embodiment, the query operation for querying metadata is the inverse operation of the process of generating the corresponding table during database sharding.

[0063] Step 23: Write the physical storage information as the access address parameter into the preset query instruction to generate a query request.

[0064] In this embodiment, the actual physical storage information corresponding to the target database table shard is determined, that is, in the multiple shards generated by the database sharding and table partitioning operation, the target database table shard is identified, and a query instruction pointing to the target database table shard is generated, thereby obtaining a query request that can query and access the target metadata.

[0065] Steps 21 to 23 above involve detecting the database table name of the target database table shard corresponding to the target metadata in the target metadata; generating physical storage information corresponding to the target database table shard based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards; writing the physical storage information as an access address parameter into a preset query instruction to generate a query request, parsing the data access request and rewriting the query request, enabling the database sharding intermediate controller to quickly query the target database table shard and complete the query and access to the target metadata, so as to return the access results to the target MetaStore instance, thereby improving the speed of metadata query.

[0066] In some embodiments, the physical storage information corresponding to the target database table shard is generated based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards. This is achieved through the following steps:

[0067] Step 31: Obtain the first logical table number parameter table, which includes the library table name, logical table number, and the correspondence between the two.

[0068] In this embodiment, the first logical table label parameter table is a mapping metadata, corresponding database table shards, and database sharding relationship table generated during the process of vertically and horizontally partitioning (database sharding and table partitioning) the metadata storage of Hive's metadata database so that each MetaStore instance corresponds to multiple database table shards.

[0069] Step 32: In the first logical table number parameter table, query the logical table number corresponding to the library table name that is sharded with the target library table.

[0070] In this embodiment, the first logical table number parameter table can be queried by traversal to determine the logical table number corresponding to the database table name of the target database table shard.

[0071] Step 33: Using the table sharding operation algorithm corresponding to the preset sharding rules, perform mathematical operations on the logical table number and the number of table shards, and determine the physical storage information corresponding to the target database table shards based on the operation values ​​obtained from the mathematical operations.

[0072] In this embodiment, a modulo operation is performed using the logical table number and the number of table shards to quickly determine the physical storage information corresponding to the target database table shard in the database table shards generated by the hash modulo operation.

[0073] The process involves obtaining the first logical table number parameter table as described above; querying the logical table number corresponding to the database table name of the target database table shard in the first logical table number parameter table; using the table sharding operation algorithm corresponding to the preset sharding rules, performing mathematical operations on the logical table number and the number of table shards, and determining the physical storage information corresponding to the target database table shard based on the operation value obtained from the mathematical operations. This process converts the logical table name into the physical table name that actually stores the target metadata, enabling the database sharding intermediate controller to quickly query the target database table shard after rewriting the query request, and complete the query and access to the target metadata, so as to return the access results to the target MetaStore instance and improve the speed of metadata query.

[0074] In some embodiments, the sharding rules include hash modulo sharding, which involves performing mathematical operations on the logical table number and the number of table shards, and determining the physical storage information corresponding to the target database table shard based on the results of the mathematical operations. This is achieved through the following steps:

[0075] Step 331: Process the logical table numbers using a hash algorithm to obtain hash values;

[0076] Step 332: Perform a modulo operation between the hash value and the number of table shards to obtain the calculated value;

[0077] Step 333: Among multiple database table shards, select the database table shard corresponding to the arrangement number and operation value of the database table, determine that the target database table shard includes the database table shard corresponding to the arrangement number and operation value, and determine that the physical storage information includes at least the arrangement number corresponding to the target database table shard.

[0078] The above steps involve processing the logical table number using a hash algorithm to obtain a hash value; performing a modulo operation between the hash value and the number of table shards to obtain an operation value; selecting the database table shard corresponding to the sequence number and operation value from multiple database table shards; determining that the target database table shard includes the database table shard corresponding to the sequence number and operation value; and ensuring that the physical storage information includes at least the sequence number corresponding to the target database table shard. This further enables the conversion of logical table names into physical table names that actually store the target metadata. This allows the sharding intermediate controller to quickly query the target database table shard after rewriting the query request, complete the query and access to the target metadata, and return the access results to the target MetaStore instance, thus improving the speed of metadata querying.

[0079] In some optional implementations, when the read / write operation corresponding to accessing metadata is a query operation, and processing is performed in the middle of database sharding, the SQL corresponding to the data access request is first parsed according to the previously configured sharding rules, the logical table name is converted into the physical table name in actual storage, and finally the query request SQL is rewritten. The query request SQL is executed via JDBC, and the metadata access result is returned. Specifically, taking the query of the TBLS table as an example, and the table code ID of the database table is denoted as idtable_id, when the query SQL is: select * from tabls where table_id = 10, at this time, tabls is a logical table, but a table with that name does not exist physically. After sharding calculation, the query request SQL is rewritten as: select * from tabls_1 where table_id = 10, where tabls_1 is the actual physical table stored in MySQL. When the read / write operation corresponding to accessing metadata is an insert operation, because it involves operations such as auto-incrementing primary keys. To address this, the sharding and partitioning intermediate controller is configured with a Twitter-Snowflake algorithm to generate globally unique 64-bit auto-incrementing IDs, configuring a globally unique ID for each primary key insertion with auto-incrementing primary keys.

[0080] In some embodiments, before acquiring the data access request forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, the following steps are also performed:

[0081] Step 41: After receiving a data access request from the client, Waggle Dance detects the target information in the data access request.

[0082] Step 42: Waggle Dance detects the target MetaStore instance based on the meta target information among multiple MetaStore instances that are connected to Waggle Dance. Each MetaStore instance is associated with a corresponding meta database.

[0083] Step 43: Waggle Dance forwards the data access request to the target MetaStore instance.

[0084] In this embodiment, when a client wants to access the corresponding metadata, it accesses Waggle Dance to fulfill the need to access the Metastore instance, so as to further query or access the target metadata through the Metastore instance. Of course, in this embodiment, querying or accessing the target metadata through the Metastore instance is converted to execution through the sharding and partitioning intermediate controller.

[0085] In this embodiment, Waggle Dance routes the data access request from the corresponding client to the corresponding Metastore instance. The target metadata carried in the data access request maps the target object (a certain Metastore instance) to which the data access request needs to be routed. Therefore, after obtaining the target metadata from the data access request, Waggle Dance can quickly determine the corresponding target Metastore instance, that is, detect the target MetaStore instance based on the target metadata. Of course, a Metastore instance corresponds to a metadata database. At the same time, by configuring the Metastore instance as a primary Metastore instance and a secondary Metastore instance, the corresponding MySQL database is also a primary / secondary MySQL database.

[0086] Waggle Dance implements the following steps: upon receiving a data access request from a client, it detects the target metadata within the request; among multiple MetaStore instances connected to Waggle Dance, it identifies the target MetaStore instance based on the metadata; and it forwards the data access request to the target MetaStore instance, thus enabling the forwarding of data access requests to different Metastores for processing.

[0087] It should be noted that in this embodiment, the execution entity for steps 41 to 43 is Waggle Dance. Waggle Dance forwards the data access request to the corresponding MetaStore instance according to the request type corresponding to the data access request submitted by the client, so that the corresponding MetaStore instance expects to send the data access request to the associated metadata database to obtain the target metadata. That is, the data access request submitted by the client needs to be routed and forwarded through Waggle Dance before it can be transmitted to the corresponding MetaStore instance. Furthermore, when the corresponding MetaStore instance expects to send the data access request to the associated metadata database, the execution entity of this application embodiment, the database sharding and table sharding intermediate controller, intercepts and obtains the corresponding data access request, and executes the metadata access method of this application embodiment based on the intercepted data access request.

[0088] In some embodiments, Waggle Dance implements the following steps to detect the target MetaStore instance based on meta-target information among multiple MetaStore instances interfaced with Waggle Dance:

[0089] Step 51: Detect the data name of the target metadata in the target metadata;

[0090] Step 52: Obtain the first configuration parameter table, wherein the first configuration parameter table includes the correspondence information between the data name corresponding to the metadata and the MetaStore instance configured for each piece of metadata;

[0091] Step 53: In the first configuration parameter table, query the MetaStore instance corresponding to the data name of the target metadata to obtain the first MetaStore instance, and determine that the target MetaStore instance includes the first MetaStore instance.

[0092] In some embodiments, the acquisition of data access requests forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance is achieved through the following steps:

[0093] Step 61: When the target MetaStore instance is about to send an intention request to the corresponding metadata database to query metadata, intercept the intention request, which carries a data access request.

[0094] Step 62: Parse the data access request from the intercepted intent request.

[0095] In this embodiment, when a MetaStore instance wants to communicate with the metadata database MySQL, the sharding and partitioning intermediate controller will intercept the intention request carrying the data access request, route the intention request to its own node, and thus parse and obtain the data access request.

[0096] In some embodiments, based on a query request, the target database table shard is queried from multiple database table shards storing metadata, so as to obtain the target metadata after accessing the target database table shard. This is achieved through the following steps: querying the target database table shard from multiple database table shards via Java Database Connectivity (JDBC), and accessing the target database table shard to obtain the target metadata.

[0097] Figure 2 This is a schematic diagram of the interaction between Waggle_dance and Hive MetaStore in an embodiment of this application. Figure 1 , Figure 3 This is a schematic diagram of the interaction between Waggle_dance and Hive MetaStore in an embodiment of this application. Figure 2 , Figure 4 This is a schematic diagram of the library segmentation in an embodiment of this application. Figure 5 This is a schematic diagram of the metadata access process in a preferred embodiment of this application. Figure 6 This is a schematic diagram of the sharding and table partitioning intermediate controller according to an embodiment of this application. The following references... Figures 2 to 6 The process and related architecture of metadata access involved in the embodiments of this application are described as follows:

[0098] refer to Figure 2 This application employs federation of the MetaStore. Specifically, it uses Hive DB for partitioning, distributing metadata across multiple MySQL environments at the HiveDB level, with each metastore instance corresponding to its own storage. A route is added between the Hiveserver2 service and the Metastore service. HiveServer2 does not directly access the Metastore, but instead directly accesses the routing layer, which forwards the requests to multiple MetaStore instances.

[0099] refer to Figure 3Waggle Dance acts as a routing service for the Metastore, implementing the Metastore's Thrift API. Users access the Metastore directly through Waggle Dance, which connects to multiple Hive Metastores, combining these previously unconnected entities into a unified whole. In this embodiment, multiple master Metastores and multiple slave Metastores are configured. The master Metastores are read-write, while the slave Metastores are read-only. This master-slave separation reduces the access pressure on the metadata database, and the multiple instances of both master and slave Metastores ensure service stability. In this embodiment, Waggle Dance can forward requests to different Metastores for processing based on the request type.

[0100] refer to Figures 4 to 5 The database sharding process in this embodiment is as follows: First, the entire MetaStore is vertically partitioned, and a database is split into multiple databases based on the closeness of table relationships, and distributed across different nodes. However, vertical partitioning alone cannot solve the problem of slow queries caused by excessive data volume in a single table, so large tables also need to be horizontally partitioned. In this embodiment, taking the TBLS table as an example, a hash partitioning method is used. The table ID is hashed and then moduloed with the number of table shards to distribute the data evenly across different tables. For example, if table_id = 10, number of shards shared_num = 3, and 10 mod 3 = 1, then the data of this table should be stored in table1.

[0101] refer to Figures 2 to 6 The metadata access process involved in this application is as follows:

[0102] The main process remains unchanged, and you can refer to steps one through six of the existing metadata access process mentioned above. The change is that when the MetaStore service wants to communicate with the metadata database MySQL, the sharding and partitioning middle controller will intercept its data access request, route the intercepted data access request to its own node, perform sharding calculation, query the database, and finally return the data to MetaStore.

[0103] During a query operation, the sharding and partitioning intermediate controller first parses the SQL according to the previously configured sharding rules, converts the logical table name into the physical table name on the actual storage, and finally rewrites the SQL, executes it via JDBC, and returns the result.

[0104] Taking querying the TBLS table as an example, the query SQL is:

[0105] select * from tabls where table_id = 10

[0106] At this point, `tabls` is a logical table, but a table with that name does not physically exist. After sharding and computation, the SQL is rewritten as follows:

[0107] select * from tabls_1 where table_id = 10

[0108] At this point, tabls_1 is the actual physical table stored in MySQL.

[0109] In some of the alternative implementations, an insertion operation is also performed. Since the insertion operation process involves operations such as auto-incrementing primary keys, the sharding and partitioning intermediate controller is configured with a Twitter-Snowflake algorithm to generate a globally unique 64-bit auto-incrementing ID, and a globally unique ID is configured for each primary key auto-incrementing insertion operation.

[0110] This embodiment also provides a metadata access device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0111] Figure 7 This is a structural block diagram of a metadata access device according to an embodiment of this application, such as... Figure 7 As shown, the metadata access device includes:

[0112] The acquisition module 71 is used to acquire data access requests forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, wherein the data access request carries the meta target information of the target metadata requested by the client.

[0113] The processing module 72, coupled to the acquisition module 71, is used to parse the metadata target information from the data access request and reconstruct the query request for querying the target database table shards that store the target metadata based on the metadata target information.

[0114] The query module 73, coupled to the processing module 72, is used to query the target database table shard from multiple database table shards storing metadata according to the query request, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to preset sharding rules.

[0115] The metadata access device in this application embodiment acquires a data access request forwarded by a preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance. The data access request carries the target metadata metadata requested by the client. The target metadata is parsed from the data access request, and a query request for querying the target database table shards storing the target metadata is reconstructed based on the target metadata. According to the query request, the target database table shard is queried from multiple database table shards storing metadata, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to preset sharding rules. This solves the problems of high query pressure, slow metadata access speed, and poor Hive stability when the Hive metadata database data storage volume is large, and achieves the beneficial effects of decompressing the metadata database, improving Hive stability, and improving the speed of metadata query and access.

[0116] In some embodiments, the processing module 72 further includes:

[0117] The first detection unit is used to detect the database table name of the target database table fragment corresponding to the target metadata in the metadata;

[0118] The first generation unit, coupled to the first detection unit, is used to generate physical storage information corresponding to the target database table shard based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards. The physical storage information is used to characterize the storage location of the target database table shard in the metadata database corresponding to the target MetaStore instance.

[0119] The first writing unit, coupled to the first generating unit, is used to write physical storage information as an access address parameter into a preset query instruction to generate a query request.

[0120] In some embodiments, the first generation unit is further configured to: obtain a first logical table number parameter table, wherein the first logical table number parameter table includes a library table name, a logical table number, and the correspondence information between the two; query the first logical table number parameter table for the logical table number corresponding to the library table name of the target library table shard; perform mathematical operations on the logical table number and the number of table shards using a table sharding operation algorithm corresponding to a preset sharding rule, and determine the physical storage information corresponding to the target library table shard based on the operation value obtained from the mathematical operations.

[0121] In some embodiments, the sharding rules include hash modulo sharding. The first generation unit is further configured to process the logical table number using a hash algorithm to obtain a hash value; perform a modulo operation on the hash value and the number of table shards to obtain an operation value; select the database table shard corresponding to the arrangement number and the operation value among multiple database table shards; determine that the target database table shard includes the database table shard corresponding to the arrangement number and the operation value; and determine that the physical storage information includes at least the arrangement number corresponding to the target database table shard.

[0122] In some embodiments, before acquiring the data access request forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, the metadata access device, through WaggleDance, detects the target metadata information in the data access request after receiving it from the client; among the multiple MetaStore instances interfaced with Waggle Dance, the target MetaStore instance is detected based on the target metadata information, wherein each MetaStore instance is associated with a corresponding metadata database; and the data access request is forwarded to the target MetaStore instance.

[0123] In some embodiments, Waggle Dance is also used to detect the data name of the target metadata in the meta target information; obtain a first configuration parameter table, wherein the first configuration parameter table includes the correspondence information between the data name corresponding to the metadata and the MetaStore instance configured for each piece of metadata; query the MetaStore instance corresponding to the data name of the target metadata in the first configuration parameter table, obtain the first MetaStore instance, and determine that the target MetaStore instance includes the first MetaStore instance.

[0124] In some embodiments, the acquisition module 71 is further configured to intercept the intent request when the target MetaStore instance is expected to send an intent request to the corresponding metadata database to query metadata, wherein the intent request carries a data access request; and parse the intercepted intent request to obtain the data access request.

[0125] In some embodiments, the query module 73 is also used to query the target database table shard across multiple database table shards via a Java Database Connectivity (JDBC) connection, and to access the target database table shard to obtain target metadata.

[0126] Figure 8 This is a schematic diagram of the electronic device structure according to an embodiment of this application, such as... Figure 8 As shown in the figure, this application provides an electronic device, including a processor 81, a communication interface 82, a memory 83, and a communication bus 84, wherein the processor 81, the communication interface 82, and the memory 83 communicate with each other through the communication bus 84.

[0127] Memory 83 is used to store computer programs;

[0128] When processor 81 executes the program stored in memory 83, it implements... Figure 1 The methods and steps in the text.

[0129] The processing implementation in this electronic device Figure 1 The method steps described above, and the resulting technical effects, are similar to those achieved in the embodiments described above. Figure 1 The technical effects of the metadata access methods are the same, so they will not be elaborated here.

[0130] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0131] The communication interface is used for communication between the aforementioned terminal and other devices.

[0132] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0133] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0134] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the metadata access method provided in any of the foregoing method embodiments.

[0135] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the metadata access methods described in the above embodiments.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for accessing metadata, characterized in that, include: Obtain a data access request forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, wherein the data access request carries the meta target information of the target metadata requested by the client; The metadata target information is parsed from the data access request, and the query request for querying the target database table shard storing the target metadata is reconstructed based on the metadata target information. According to the query request, the target database table shard is queried from multiple database table shards that store metadata, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to a preset sharding rule. The process of reconstructing the query request for the target database table shard storing the target metadata based on the meta-target information includes: Detect the database table name of the target database table shard corresponding to the target metadata in the target metadata; Based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards, physical storage information corresponding to the target database table shard is generated, wherein the physical storage information is used to characterize the storage location of the target database table shard in the metadata database corresponding to the target MetaStore instance; The physical storage information is written as an access address parameter into a preset query instruction to generate the query request; Specifically, based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards, the physical storage information corresponding to the target database table shard is generated, including: Obtain the first logical table number parameter table, wherein the first logical table number parameter table includes the library table name, the logical table number, and the correspondence information between the two; In the first logical table number parameter table, query the logical table number corresponding to the library table name that is sharded with the target library table; Using the table sharding operation algorithm corresponding to the preset sharding rules, mathematical operations are performed on the logical table number and the number of table shards, and the physical storage information corresponding to the target database table shard is determined based on the operation value obtained from the mathematical operations.

2. The method according to claim 1, characterized in that, The sharding rules include hash modulo sharding, which involves performing mathematical operations on the logical table number and the number of table shards, and determining the physical storage information corresponding to the target database table shard based on the calculated value, including: The logical table numbers are processed using a hash algorithm to obtain hash values; The hash value is moduloed by the number of table shards to obtain the calculated value; Among the multiple database table shards, the database table shard corresponding to the arrangement number of the database table and the operation value is selected, and the target database table shard is determined to include the database table shard corresponding to the arrangement number and the operation value. The physical storage information is determined to include at least the arrangement number corresponding to the target database table shard.

3. The method according to claim 1, characterized in that, Before acquiring the data access request forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, the method includes: After receiving the data access request sent by the client, Waggle Dance detects the meta-target information in the data access request; In the multiple MetaStore instances that are connected to Waggle Dance, the target MetaStore instance is detected based on the meta target information, wherein each MetaStore instance is associated with a corresponding meta database; The Waggle Dance forwards the data access request to the target MetaStore instance.

4. The method according to claim 3, characterized in that, The Waggle Dance, among multiple MetaStore instances that interface with it, detects the target MetaStore instance based on the meta-target information, including: The Waggle Dance detects the data name of the target metadata in the meta-target information; Waggle Dance obtains a first configuration parameter table, wherein the first configuration parameter table includes the correspondence information between the data name corresponding to the metadata and the MetaStore instance configured for each piece of metadata; Waggle Dance queries the first configuration parameter table to find the MetaStore instance corresponding to the data name of the target metadata, obtains the first MetaStore instance, and determines that the target MetaStore instance includes the first MetaStore instance.

5. The method according to claim 3, characterized in that, Retrieve data access requests forwarded by the pre-defined federated routing management server WaggleDance to the target metadata storage MetaStore instance, including: When the target MetaStore instance is about to send an intention request to the corresponding metadata database to query metadata, the intention request is intercepted, wherein the intention request carries the data access request; The data access request is obtained by parsing the intercepted intent request.

6. The method according to claim 1, characterized in that, Based on the query request, the target database table shard is queried from multiple database table shards storing metadata, so as to obtain the target metadata after accessing the target database table shard. This includes: querying the target database table shard from multiple database table shards via Java Database Connectivity (JDBC), and accessing the target database table shard to obtain the target metadata.

7. A metadata access device, characterized in that, include: The acquisition module is used to acquire data access requests forwarded by the preset federated routing management server Waggle Dance to the target metadata storage MetaStore instance, wherein the data access request carries the meta target information of the target metadata requested by the client; The processing module is used to parse the metadata target information from the data access request and reconstruct the query request for querying the target database table shard that stores the target metadata based on the metadata target information. The access module is used to query the target database table shard from multiple database table shards storing metadata according to the query request, so as to obtain the target metadata after accessing the target database table shard. The multiple database table shards are generated by splitting the metadata database corresponding to the target MetaStore instance according to a preset sharding rule. The processing module includes: The first detection unit is used to detect the database table name of the target database table fragment corresponding to the target metadata in the metadata; The first generation unit, coupled to the first detection unit, is used to generate physical storage information corresponding to the target database table shard based on the logical table number corresponding to the database table name of the target database table shard and the number of table shards corresponding to multiple database table shards. The physical storage information is used to characterize the storage location of the target database table shard in the metadata database corresponding to the target MetaStore instance. The first writing unit, coupled to the first generating unit, is used to write physical storage information as an access address parameter into a preset query instruction to generate a query request. The first generation unit is further configured to: obtain a first logical table number parameter table, wherein the first logical table number parameter table includes a library table name, a logical table number, and the correspondence information between the two; query the first logical table number parameter table for the logical table number corresponding to the library table name of the target library table shard; perform mathematical operations on the logical table number and the number of table shards using a table sharding operation algorithm corresponding to a preset sharding rule, and determine the physical storage information corresponding to the target library table shard based on the operation value obtained from the mathematical operations.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the metadata access method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the metadata access method as described in any one of claims 1-6.

Images