Hive-based metadata forwarding method and device, storage medium and electronic equipment

Abstract

The invention relates to a Hive-based metadata forwarding method and device, a storage medium and electronic equipment. The method comprises the steps that under the condition that a storage instruction is received, a target library name of target metadata to be stored is obtained from the storage instruction through a target class, the target library name is the name of a database where the target metadata is to be stored, and the target class is the class which is added into Hive and used for forwarding the target metadata; determining a target metadata forwarding component from the plurality of metadata forwarding components through a corresponding relation between the target class query library name and the metadata forwarding components; and sending the target metadata to the target metadata forwarding component, so that the target metadata forwarding component sends the target metadata to the target read-write component. According to the method and the device, the technical problem that the bottleneck exists in the Hive metadata storage is solved.

Description

Technical Field

[0001] This application relates to the field of data storage, and in particular, to a method, apparatus, storage medium, and electronic device for forwarding metadata based on Hive. Background Art

[0002] Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. Hadoop is a distributed architecture that provides program development services for users unfamiliar with distributed systems.

[0003] In the prior art, the metadata of Hive needs to be stored in a library of the underlying database mysql. Using a single mysql library to store the metadata of Hive may result in insufficient storage capacity of the mysql library. Summary of the Invention

[0004] This application provides a method, apparatus, storage medium, and electronic device for forwarding metadata based on Hive to solve the technical problem of bottlenecks in the storage of Hive metadata.

[0005] In a first aspect, this application provides a method for forwarding metadata based on Hive, including: when a storage instruction is received, obtaining the target library name of the target metadata to be stored from the storage instruction through a target class, where the target library name is the name of the database to which the target metadata is to be stored, the target class is a class added to the Hive for forwarding the target metadata, and the target class declares an interface of a metadata forwarding component; querying the correspondence between the library name and the metadata forwarding component through the target class to determine a target metadata forwarding component from multiple metadata forwarding components; and sending the target metadata to the target metadata forwarding component, so that the target metadata forwarding component sends the target metadata to a target read / write component, where the target read / write component corresponds to the target metadata forwarding component, and the target read / write component is used to store the target metadata in a target database corresponding to the target read / write component.

[0006] Second aspect, the present application provides a metadata forwarding device based on Hive, including: an acquisition module, configured to, when receiving a storage instruction, obtain the target library name of the target metadata to be stored from the above storage instruction through a target class, where the above target library name is the name of the database to which the above target metadata is to be stored, the above target class is a class added to the above Hive for forwarding the above target metadata, and the above target class declares an interface of a metadata forwarding component; a query module, configured to query the correspondence between the library name and the metadata forwarding component through the above target class to determine a target metadata forwarding component from multiple metadata forwarding components; a sending module, configured to send the above target metadata to the above target metadata forwarding component, so that the above target metadata forwarding component sends the above target metadata to a target read / write component, where the above target read / write component corresponds to the above target metadata forwarding component, and the above target read / write component is configured to store the above target metadata into a target database corresponding to the above target read / write component.

[0007] As an optional example, the above device further includes: a database partitioning module, configured to, before receiving the above storage instruction, partition multiple databases for the above metadata according to the type of the metadata to be stored, where the above target metadata is a piece of data in the above metadata to be stored; assign a library name to each of the above databases; construct a read / write component and a metadata forwarding component for each of the above databases, where one of the above databases corresponds to one of the above read / write components, and one of the above read / write components corresponds to one of the above metadata forwarding components.

[0008] As an optional example, the above database partitioning module includes: a first database partitioning unit, configured to determine the library or table to which the database data corresponding to the above metadata belongs; partition the metadata of the database data belonging to the same library or the same table into one type of metadata; and partition each type of metadata into a database.

[0009] As an optional example, the above database partitioning module includes: a second database partitioning unit, configured to determine the field type of the database data corresponding to the above metadata; partition the metadata corresponding to the database data with the same above field type into one type of metadata; and partition each type of metadata into a database.

[0010] As an optional example, the above acquisition module includes: a processing unit, configured to, when receiving the above storage instruction, determine the current processing strategy; when the above current processing strategy is to perform a database partitioning operation on the above target metadata, obtain the above target library name; and when the above current processing strategy is not to perform a database partitioning operation on the above target metadata, store the above target metadata into a default database.

[0011] As an alternative example, the above query module includes: a determination unit configured to pass the above target library name into the correspondence between the above library name and the metadata forwarding component; and determine the metadata forwarding component generated after passing the above target library name as the above target metadata forwarding component.

[0012] As an alternative example, the above device further includes: a control module configured to, after sending the above target metadata to the above target metadata forwarding component to enable the above target metadata forwarding component to send the above target metadata to the target read / write component, control the above target read / write component to perform at least one of the operations of permission verification operation, flow limiting operation, or auditing operation on the above storage instruction.

[0013] In a third aspect, the present application provides an electronic device, including: at least one communication interface; at least one bus connected to the above at least one communication interface; at least one processor connected to the above at least one bus; and at least one memory connected to the above at least one bus, wherein the above memory stores a computer program, and the above processor is configured to implement the above-mentioned Hive-based metadata forwarding method according to any one of the above when executing the above computer program.

[0014] In a fourth aspect, the present application further provides a computer storage medium storing computer-executable instructions, and the above computer-executable instructions are used to execute the above-mentioned Hive-based metadata forwarding method according to any one of the above in the present application.

[0015] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: In the solution provided by the embodiments of the present application, when a storage instruction is received, the target library name of the target metadata to be stored is obtained, and then, according to the correspondence between the library name and the metadata forwarding component, the target metadata forwarding component is determined, the target metadata is sent to the target metadata forwarding component, the target metadata forwarding component sends the target metadata to the corresponding target read / write component, and the target read / write component stores the target metadata into the corresponding target database, thereby storing the metadata of Hive in separate libraries and solving the problem of storage bottleneck in a single library. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated. The drawings in the figures do not constitute a scale limitation.

[0019] Figure 1 It is a flowchart of a metadata forwarding method based on Hive provided by an embodiment of the present application;

[0020] Figure 2 It is a metadata management component diagram of a metadata forwarding method based on Hive provided by an embodiment of the present application;

[0021] Figure 3 It is a flowchart of another metadata forwarding method based on Hive provided by an embodiment of the present application;

[0022] Figure 4 It is a mapping relationship diagram of a metadata forwarding method based on Hive provided by an embodiment of the present application;

[0023] Figure 5 It is a flowchart of yet another metadata forwarding method based on Hive provided by an embodiment of the present application;

[0024] Figure 6 It is a flowchart of yet another metadata forwarding method based on Hive provided by an embodiment of the present application;

[0025] Figure 7 It is a processing schematic diagram of whether to split databases for a metadata forwarding method based on Hive provided by an embodiment of the present application;

[0026] Figure 8 It is a metadata forwarding schematic diagram of a metadata forwarding method based on Hive provided by an embodiment of the present application;

[0027] Figure 9 It is a structural schematic diagram of a metadata forwarding device based on Hive provided by an embodiment of the present application;

[0028] Figure 10 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some but not all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0030] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplification and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0031] To solve the technical problem of the bottleneck in the metadata storage of Hive in the prior art, this application provides a metadata forwarding method based on Hive, which can solve the problem of the storage bottleneck in a single database.

[0032] Figure 1 The flowchart of a metadata forwarding method based on Hive provided for the embodiments of this application. As Figure 1 shown, the above-mentioned metadata forwarding method based on Hive includes:

[0033] S102, when a storage instruction is received, obtain the target database name of the target metadata to be stored from the storage instruction through the target class, where the target database name is the name of the database to which the target metadata is to be stored, the target class is the class added to Hive for forwarding the target metadata, and the target class declares the interface of the metadata forwarding component;

[0034] S104, query the correspondence between the database name and the metadata forwarding component through the target class to determine the target metadata forwarding component from multiple metadata forwarding components;

[0035] S106, send the target metadata to the target metadata forwarding component so that the target metadata forwarding component sends the target metadata to the target read / write component, where the target read / write component corresponds to the target metadata forwarding component, and the target read / write component is used to store the target metadata into the target database corresponding to the target read / write component.

[0036] In this embodiment, the above metadata is the metadata of Hive. Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. Hadoop is a distributed architecture that provides program development services for users who are not familiar with distributed systems. Hive metadata: Metadata is data that describes data. For example, the size of a data table is 100KB. The data table is data, and the table size is the data of the data. Hive metadata includes hive library information, table information (table attributes, table name, table columns, partitions and their attributes), and the directory where the table data is located, etc.

[0037] Hive metadata can be managed by the service of Hive: the metadata management component Hive MetaStore. The HiveMetaStore includes the metadata forwarding component HMSHandler component and the read / write component ObjectStore component. Among them, HMSHandler is responsible for forwarding the metadata to ObjectStore, and ObjectStore is responsible for reading and writing to the underlying mysql database. ObjectStore and the underlying mysql database are in a one-to-one relationship, that is, one ObjectStore reads and writes to one mysql database.

[0038] In this embodiment, the target metadata is the metadata in Hive metadata that the storage instruction indicates to store. For the target metadata, first, it is necessary to determine which library to store it in. That is, check the target library name of the target metadata. Querying the target library name is completed by the target class. The target class is a class newly added to Hive MetaStore in this embodiment. The class declaration has the interface of the above metadata forwarding component HMSHandler, and this interface can be called to forward the target metadata to ObjectStore for storage in the database by ObjectStore.

[0039] The target class stores the mapping relationship between the library name and HMSHandler. When the target library name is obtained from the storage instruction and input into this mapping relationship, the corresponding HMSHandler can be obtained. With the HMSHandler, the target metadata can be forwarded to ObjectStore, and ObjectStore stores the target metadata in the corresponding database.

[0040] Figure 2 It is a schematic diagram of the metadata management component in this embodiment. The target class is added to the metadata management component. The target class has the mapping relationship between the library name and the metadata forwarding component. The metadata forwarding component is determined according to the mapping relationship, and finally the read / write component writes the metadata into the database.

[0041] In the solution provided by the embodiment of this application, when a storage instruction is received, the target library name of the target metadata to be stored is obtained. Then, according to the correspondence between the library name and the metadata forwarding component, the target metadata forwarding component is determined, and the target metadata is sent to the target metadata forwarding component. The target metadata forwarding component sends the target metadata to the corresponding target read / write component, and the target read / write component stores the target metadata in the corresponding target database, thereby storing the metadata of Hive in multiple libraries and solving the problem of storage bottlenecks in a single library.

[0042] As an alternative example, as Figure 3 shown, before receiving the storage instruction, the above method further includes:

[0043] S302, divide the metadata into multiple databases according to the type of the metadata to be stored, where the target metadata is a piece of data in the metadata to be stored;

[0044] S304, assign a library name to each database;

[0045] S306, construct a read / write component and a metadata forwarding component for each database, where one database corresponds to one read / write component, and one read / write component corresponds to one metadata forwarding component.

[0046] In this embodiment, the mapping relationship between the above library name and the metadata forwarding component needs to be set in advance. According to the type of the metadata to be stored in Hive, the metadata is classified to obtain multiple types of metadata, and each type of metadata is stored in one database. Therefore, each type of metadata should be assigned a library name. When any metadata in this type of metadata is to be stored in the database as the target metadata, the library name is used to determine the metadata forwarding component.

[0047] Each type of metadata corresponds to a metadata forwarding component, that is, the HMSHandler component. When any metadata in any type of metadata is to be stored in the database as the target metadata, the HMSHandler component can be mapped according to the library name.

[0048] Figure 4 This is a schematic diagram of the mapping relationship between the library name and the metadata forwarding component in this embodiment. One library name corresponds to one metadata forwarding component, the library names are not repeated, and the metadata forwarding components are not repeated.

[0049] As an alternative example, as Figure 5 shown, dividing the metadata into multiple databases according to the type of the metadata to be stored includes:

[0050] S502, determine the library or table to which the database data corresponding to the metadata belongs;

[0051] S504. Classify the metadata of database data belonging to the same library or the same table into one type of metadata;

[0052] S506. Divide each type of metadata into a database.

[0053] This embodiment is a method for classifying Hive metadata, that is, classifying metadata according to the library or table to which the data corresponding to the metadata belongs. For the database data corresponding to Hive metadata, the metadata corresponding to the data located in the same database or table is classified into the same type of metadata. Or, for tables and libraries storing the same fields, their metadata is classified into one type of metadata.

[0054] For example, if both database 1 and database 2 store the name field, then the metadata of the two databases can be classified into one type of metadata. Specifically, it can also be determined whether to classify the metadata of different databases into one type of metadata according to the quantity and proportion of the same resources included in the databases.

[0055] As an optional example, as Figure 6 shown, dividing multiple databases for metadata according to the type of metadata to be stored includes:

[0056] S602. Determine the field type of the database data corresponding to the metadata;

[0057] S602. Classify the metadata corresponding to the database data with the same field type into one type of metadata;

[0058] S602. Divide each type of metadata into a database.

[0059] In this embodiment, the field type of the database data can be determined, and the type of metadata can be divided according to the fields. For example, for the same field located in different databases, the metadata corresponding to this field is classified into one type, and the metadata corresponding to other fields is classified into another type.

[0060] For example, if database 1 includes the name and gender fields, and database 2 includes the name and age fields, then the metadata corresponding to the name field in database 1 and the name field in database 2 is classified into one type, the metadata corresponding to the gender field in database 1 is classified into another type, and the metadata corresponding to the age field in database 2 is classified into the third type.

[0061] As an alternative example, when a storage instruction is received, the target library name of the target metadata to be stored obtained from the storage instruction by the target class includes: when a storage instruction is received, determining the current processing strategy; when the current processing strategy is to perform database sharding on the target metadata, obtaining the target library name; when the current processing strategy is not to perform database sharding on the target metadata, storing the target metadata in the default database.

[0062] In this embodiment, a default database can also be set up. The default database is a database used to store metadata that does not require database sharding. In this embodiment, when a storage instruction is received, first check whether the target metadata to be stored in the storage instruction needs to be sharded. If it does not need to be sharded, use the library name of the default database as the library name of the target metadata and directly store the target metadata in the default database. If database sharding is to be performed on the target metadata, determine the metadata forwarding tool according to the library name of the target metadata and perform the database sharding operation.

[0063] Whether to perform database sharding on the target metadata can be determined by the current processing strategy. The current processing strategy can be a specific field. For example, 0 represents no sharding, and 1 represents sharding.

[0064] Figure 7 This is the schematic diagram of whether to perform database sharding in this embodiment. The processing strategy parameter hive.metastore.hmshandler.type can be set. Different values of the parameter determine whether to use the database sharding function. If the database sharding function is not used, for the target metadata, the original HMSHandler can be directly used to store the metadata in the default database. If the database sharding function is used, a Handler of the DBRouterHMSHandler type is added, and the metadata requested by different Hive library names (DBName) is stored in different Mysql DBName databases.

[0065] As an alternative example, querying the correspondence between the library name and the metadata forwarding component through the target class to determine the target metadata forwarding component from multiple metadata forwarding components includes: passing the target library name into the correspondence between the library name and the metadata forwarding component; determining the metadata forwarding component generated after passing in the target library name as the target metadata forwarding component.

[0066] In this embodiment, there is a mapping relationship between the library name and the metadata forwarding component. The mapping relationship can be stored by a mapping function. When determining the metadata forwarding component, pass the library name into the mapping function, and the output of the mapping function is the metadata forwarding component corresponding to the library name. If there is no corresponding metadata forwarding component for the input library name, an error message can be output.

[0067] As an optional example, after sending the target metadata to the target metadata forwarding component so that the target metadata forwarding component sends the target metadata to the target read / write component, the above method further includes: the target read / write component performing at least one of an operation for verifying the execution permission of the storage instruction, a rate limiting operation, or an auditing operation.

[0068] In this embodiment, after the target metadata is sent to the target metadata forwarding component, the target metadata forwarding component sends the target metadata to the target read / write component, and the target read / write component can perform at least one of an operation for verifying the execution permission of the target metadata, a rate limiting operation, or an auditing operation. Figure 8 In [the above], the mapping relationship can be stored in a mapping database, and the metadata management component protection operation HiveMetaStore Ranger can be executed through the K Desktop Environment (KDE). When DBRouterHMSHandler receives a query metadata-related request, it can forward it to the lower-layer ObjectStore. Before the ObjectStore performs specific metadata operations, functions such as permission, rate limiting, and auditing are added. For example, in terms of permission, the ranger rest api can be called to determine whether the user has the operation permission, etc. If the user does not have the operation permission, the target metadata is not stored in the database or is stored in the default database.

[0069] Figure 9 It is a schematic structural diagram of a metadata forwarding device based on Hive provided by an embodiment of the present application. As Figure 9 shown, the above-mentioned metadata forwarding device based on Hive includes:

[0070] An obtaining module 902, configured to, when receiving a storage instruction, obtain the target library name of the target metadata to be stored from the storage instruction through a target class, where the target library name is the name of the database to which the target metadata is to be stored, the target class is a class added to Hive for forwarding the target metadata, and the target class declares an interface of the metadata forwarding component;

[0071] A query module 904, configured to query the correspondence relationship between the library name and the metadata forwarding component through the target class, so as to determine the target metadata forwarding component from multiple metadata forwarding components;

[0072] A sending module 906, configured to send the target metadata to the target metadata forwarding component, so that the target metadata forwarding component sends the target metadata to the target read / write component, where the target read / write component corresponds to the target metadata forwarding component, and the target read / write component is configured to store the target metadata into a target database corresponding to the target read / write component.

[0073] In this embodiment, the above metadata is the metadata of Hive. Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. Hadoop is a distributed architecture that provides program development services for users who are not familiar with distributed systems. The metadata of Hive: Metadata is data that describes data. For example, the size of a data table is 100KB. The data table is data, and the table size is the data of the data. Hive metadata includes hive library information, table information (table attributes, table name, table columns, partitions and their attributes), and the directory where the table data is located, etc.

[0074] The Hive metadata can be managed by the service of Hive: the metadata management component Hive MetaStore. The HiveMetaStore includes a metadata forwarding component HMSHandler component and a read / write component ObjectStore component. Among them, the HMSHandler is responsible for forwarding the metadata to the ObjectStore, and the ObjectStore is responsible for reading and writing to the underlying mysql database. The ObjectStore and the underlying mysql database have a one-to-one relationship, that is, one ObjectStore reads and writes to one mysql database.

[0075] In this embodiment, the target metadata is the metadata in the Hive metadata that the storage instruction indicates to store. For the target metadata, the first thing to determine is which library to store it in. That is, check the target library name of the target metadata. Querying the target library name is completed by the target class. The target class is a class newly added to the Hive MetaStore in this embodiment. The class declaration has the interface of the above metadata forwarding component HMSHandler, and this interface can be called to forward the target metadata to the ObjectStore for storage in the database by the ObjectStore.

[0076] The target class stores the mapping relationship between the library name and the HMSHandler. After obtaining the target library name from the storage instruction, input the target library name into this mapping relationship, and the corresponding HMSHandler can be obtained. With the HMSHandler, the target metadata can be forwarded to the ObjectStore, and the ObjectStore stores the target metadata in the corresponding database.

[0077] Figure 2 It is a schematic diagram of the metadata management component in this embodiment. The target class is added to the metadata management component. The target class has a mapping relationship between the library name and the metadata forwarding component. Determine the metadata forwarding component according to the mapping relationship, and finally the read / write component writes the metadata into the database.

[0078] When the solution provided by the embodiment of this application receives a storage instruction, it obtains the target library name of the target metadata to be stored. Then, according to the correspondence between the library name and the metadata forwarding component, it determines the target metadata forwarding component, sends the target metadata to the target metadata forwarding component, and the target metadata forwarding component sends the target metadata to the corresponding target read / write component, and the target read / write component stores the target metadata in the corresponding target database, thereby storing the metadata of Hive in separate libraries and solving the problem of storage bottlenecks in a single library.

[0079] For other examples of this embodiment, please refer to the above examples and will not be elaborated here.

[0080] As Figure 10 shown, the embodiment of this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 complete communication with each other through the communication bus 114.

[0081] The memory 113 is used to store a computer program.

[0082] In an embodiment of this application, when the processor 111 executes the program stored on the memory 113, it implements the metadata forwarding method based on Hive provided by any one of the foregoing method embodiments.

[0083] The embodiment of this application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the metadata forwarding method based on Hive provided by any one of the foregoing method embodiments.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution or the part that contributes to the related technology can be embodied in the form of a software product, and this computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "comprising", "including", "containing", and "having" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the particular order described or illustrated, unless the order of performance is explicitly stated. It should also be understood that additional or alternative steps may be used.

[0087] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for forwarding metadata based on Hive, characterized in that, Including: When receiving a storage instruction, obtain the target library name of the target metadata to be stored from the storage instruction through the target class, where the target library name is the name of the database to which the target metadata is to be stored, the target class is the class added to the Hive for forwarding the target metadata, and the target class declares an interface of the metadata forwarding component; Query the correspondence between the library name and the metadata forwarding component through the target class to determine the target metadata forwarding component from multiple metadata forwarding components; Send the target metadata to the target metadata forwarding component so that the target metadata forwarding component sends the target metadata to the target read / write component, where the target read / write component corresponds to the target metadata forwarding component, and the target read / write component is used to store the target metadata into the target database corresponding to the target read / write component.

2. The method according to claim 1, characterized in that, Before receiving the storage instruction, the method further includes: According to the type of the metadata to be stored, divide multiple databases for the metadata, where the target metadata is a piece of data in the metadata to be stored; Assign a library name to each of the databases; Construct a read / write component and a metadata forwarding component for each of the databases, where one database corresponds to one read / write component, and one read / write component corresponds to one metadata forwarding component.

3. The method according to claim 2, characterized in that, The dividing multiple databases for the metadata according to the type of the metadata to be stored includes: Determine the library or table to which the database data corresponding to the metadata belongs; Divide the metadata of the database data belonging to the same library or the same table into one type of metadata; Divide each type of metadata into a database.

4. The method according to claim 2, characterized in that, The dividing multiple databases for the metadata according to the type of the metadata to be stored includes: Determine the field type of the database data corresponding to the metadata; Divide the metadata corresponding to the database data with the same field type into one type of metadata; Divide each type of metadata into a database.

5. The method according to claim 1, characterized in that, The obtaining the target library name of the target metadata to be stored from the storage instruction through the target class when receiving the storage instruction includes: When receiving the storage instruction, determine the current processing strategy; When the current processing strategy is to perform database partitioning on the target metadata, obtain the target library name; When the current processing strategy is not to perform database partitioning on the target metadata, store the target metadata into the default database.

6. The method according to claim 1, characterized in that, The querying the correspondence between the library name and the metadata forwarding component through the target class to determine the target metadata forwarding component from multiple metadata forwarding components includes: Pass the target library name into the correspondence between the library name and the metadata forwarding component; Determine the metadata forwarding component generated after passing the target library name as the target metadata forwarding component.

7. The method according to claim 1, characterized in that, After sending the target metadata to the target metadata forwarding component so that the target metadata forwarding component sends the target metadata to the target read / write component, the method further includes: performing, by the target read / write component, at least one of an operation for verifying the execution permission of the storage instruction, a current limiting operation, or an auditing operation.

8. A metadata forwarding device based on Hive, characterized in that, including: an obtaining module, configured to, when receiving a storage instruction, obtain, from the storage instruction, a target library name of target metadata to be stored through a target class, where the target library name is a name of a database to which the target metadata is to be stored, the target class is a class added to the Hive for forwarding the target metadata, and the target class declares an interface of the metadata forwarding component; a querying module, configured to query, through the target class, a correspondence between the library name and the metadata forwarding component, so as to determine a target metadata forwarding component from multiple metadata forwarding components; a sending module, configured to send the target metadata to the target metadata forwarding component, so that the target metadata forwarding component sends the target metadata to a target read / write component, where the target read / write component corresponds to the target metadata forwarding component, and the target read / write component is configured to store the target metadata into a target database corresponding to the target read / write component.

9. An electronic device, characterized in that, including: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus;at least one memory connected to the at least one bus, where a computer program is stored in the memory, and when the processor executes the computer program, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions for performing the method according to any one of claims 1 to 7 of the present application.

Images