Metadata management method, device and electronic equipment

Abstract

The embodiment of the present application provides a metadata management method, device and electronic device, which relates to the field of big data technology. Based on the general Hive metadata interface service, a first data request for metadata of data to be processed in a first data source sent by a data request device is received; the first data request is formatted according to the protocol standard of the first data source to obtain a second data request; the second data request is sent to the first data source so that the first data source returns the first metadata of the data to be processed according to the second data request; according to the protocol standard supported by the data request device, based on the Hive metadata interface service, a data response containing the first metadata is sent to the data request device. Based on this, the cost of metadata management can be reduced.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a metadata management method, device and electronic device. Background Art

[0002] With the rapid development of computer technology, the amount of data is also growing rapidly. In related technologies, metadata can be used to improve the efficiency of data management. Metadata is used to describe the information of data attributes to achieve functions such as indicating the data storage location, searching for historical data, and file records.

[0003] In the related technology, a data requesting device (for example, a data operation engine) can send a metadata request to a metadata management device through a data request interface, and the metadata management device can send the requested metadata to the data requesting device. Furthermore, the data requesting device can request corresponding data from the data source based on the obtained metadata and process it.

[0004] However, since the standards of the data request interfaces of different data request devices are different, it is necessary to configure metadata management devices that are compatible with the data request interfaces of different data request devices, which will increase the cost of metadata management. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a metadata management method, device and electronic device to reduce the cost of metadata management. The specific technical solution is as follows:

[0006] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application discloses a metadata management method, the method comprising:

[0007] Based on a general Hive metadata interface service, receiving a first data request for metadata of to-be-processed data in a first data source sent by a data requesting device;

[0008] Convert the first data request into a second data request according to a protocol standard of the first data source;

[0009] Sending the second data request to the first data source, so that the first data source returns first metadata of the data to be processed according to the second data request;

[0010] According to the protocol standard supported by the data requesting device and based on the Hive metadata interface service, a data response including the first metadata is sent to the data requesting device.

[0011] Optionally, the first data source represents a structured data source; the first metadata represents a data structure of the data to be processed;

[0012] Before sending a data response including the first metadata to the data requesting device based on the Hive metadata interface service according to the protocol standard supported by the data requesting device, the method further includes:

[0013] According to the first data request, obtaining second metadata recorded locally and representing attributes of the data to be processed;

[0014] The sending a data response including the first metadata to the data requesting device based on the Hive metadata interface service according to the protocol standard supported by the data requesting device includes:

[0015] According to the protocol standard supported by the data requesting device and based on the Hive metadata interface service, a data response including the first metadata and the second metadata is sent to the data requesting device.

[0016] Optionally, the first data request carries information to be authenticated of the data requesting device;

[0017] Before sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request, the method further includes:

[0018] Based on the information to be authenticated, sending an authentication request to a designated authentication service;

[0019] The sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request includes:

[0020] When receiving the authentication success message returned by the authentication service, the second data request is sent to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request.

[0021] Optionally, before sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request, the method further includes:

[0022] Obtaining a configuration file corresponding to the first data source recorded in a distributed storage system; wherein the configuration file records connection information of the first data source;

[0023] The sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request includes:

[0024] Based on the connection information, the second data request is sent to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request.

[0025] Optionally, the method further includes:

[0026] Monitor metadata access events to obtain operation data of each data operation associated with metadata access events triggered within a historical time period;

[0027] The operation data is sent to a preset queue, so that a preset data analysis device obtains the operation data from the preset queue and performs processing based on the operation data.

[0028] Optionally, the method is applied to a metadata management node, and the method further includes:

[0029] Based on preset monitoring indicators, access status data of the data requesting device for the data to be processed and system status data of the metadata management node within a historical time period are obtained.

[0030] In a second aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application discloses a metadata management device, the device comprising:

[0031] A first data request acquisition module, configured to receive, based on a general Hive metadata interface service, a first data request for metadata of to-be-processed data in a first data source sent by a data request device;

[0032] a request conversion module, configured to convert the format of the first data request according to a protocol standard of the first data source to obtain a second data request;

[0033] a request sending module, configured to send the second data request to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request;

[0034] A response module is used to send a data response including the first metadata to the data request device based on the Hive metadata interface service in accordance with a protocol standard supported by the data request device.

[0035] Optionally, the first data source represents a structured data source; the first metadata represents a data structure of the data to be processed;

[0036] The device also includes:

[0037] A metadata acquisition module, configured to acquire, according to the first data request, second metadata representing the attributes of the data to be processed and recorded locally, before sending a data response including the first metadata to the data request device based on the Hive metadata interface service in accordance with the protocol standard supported by the data request device;

[0038] The response module is specifically configured to send a data response including the first metadata and the second metadata to the data request device based on the Hive metadata interface service in accordance with a protocol standard supported by the data request device.

[0039] Optionally, the first data request carries information to be authenticated of the data requesting device;

[0040] The device also includes:

[0041] an authentication request sending module, configured to send an authentication request to a designated authentication service based on the information to be authenticated before sending the second data request to the first data source so that the first data source returns the first metadata of the data to be processed according to the second data request;

[0042] The request sending module is specifically configured to send the second data request to the first data source upon receiving an authentication success message returned by the authentication service, so that the first data source returns the first metadata of the data to be processed according to the second data request.

[0043] Optionally, the device further comprises:

[0044] a configuration file acquisition module, configured to acquire a configuration file corresponding to the first data source recorded in a distributed storage system before sending the second data request to the first data source so that the first data source returns the first metadata of the data to be processed according to the second data request; wherein the configuration file records the connection information of the first data source;

[0045] The request sending module is specifically configured to send the second data request to the first data source based on the connection information, so that the first data source returns the first metadata of the data to be processed according to the second data request.

[0046] Optionally, the device further comprises:

[0047] A monitoring module is used to monitor metadata access events and obtain operation data of various data operations associated with metadata access events triggered within a historical time period;

[0048] The operation data sending module is used to send the operation data to a preset queue, so that a preset data analysis device obtains the operation data from the preset queue and processes the operation data based on the operation data.

[0049] Optionally, the device is applied to a metadata management node, and the device further includes:

[0050] The monitoring module is used to obtain the access status data of the data requesting device for the data to be processed and the system status data of the metadata management node within a historical time period based on preset monitoring indicators.

[0051] In another aspect of the implementation of the present application, in order to achieve the above-mentioned purpose, the embodiment of the present application further discloses an electronic device, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0052] The memory is used to store computer programs;

[0053] The processor is used to implement the metadata management method as described in the first aspect when executing the program stored in the memory.

[0054] In another aspect of the implementation of the present application, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the metadata management method as described in the first aspect is implemented.

[0055] In another aspect of the implementation of the present application, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the metadata management method described in the first aspect above.

[0056] An embodiment of the present application provides a metadata management method, which can receive a first data request for metadata of to-be-processed data in a first data source sent by a data request device based on a general Hive metadata interface service; convert the format of the first data request according to a protocol standard of the first data source to obtain a second data request; send the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request; and send a data response containing the first metadata to the data request device based on the Hive metadata interface service according to the protocol standard supported by the data request device.

[0057] The universal Hive metadata interface service can match the data request interface of different data request devices. Therefore, based on the metadata management method of the embodiment of the present application, there is no need to configure a corresponding metadata management device for each data request device, which can reduce the cost of metadata management. In addition, the data request can be converted to obtain a data request that matches the data source, which can support different types of data sources. Therefore, it can also realize the management of metadata of heterogeneous data sources and improve the efficiency of metadata management.

[0058] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A flowchart of a metadata management method provided in an embodiment of the present application;

[0061] Figure 2 A flowchart of another metadata management method provided in an embodiment of the present application;

[0062] Figure 3 A flowchart of another metadata management method provided in an embodiment of the present application;

[0063] Figure 4 A flowchart of another metadata management method provided in an embodiment of the present application;

[0064] Figure 5 An architecture diagram of each service layer in an electronic device provided in an embodiment of the present application;

[0065] Figure 6 A structural diagram of a metadata management device provided in an embodiment of the present application;

[0066] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] In the related art, since different data requesting devices have different standards for data requesting interfaces, it is necessary to configure metadata management devices compatible with their data requesting interfaces for different data requesting devices, which will increase the cost of metadata management.

[0069] In order to solve the above problems, an embodiment of the present application provides a metadata management method, which can be applied to an electronic device. In one implementation, the electronic device can be a server, which is used to manage metadata of different data sources.

[0070] See also Figure 1 , Figure 1 A flowchart of a metadata management method provided in an embodiment of the present application, the method may include the following steps:

[0071] S101: Based on a general Hive metadata interface service, receiving a first data request for metadata of to-be-processed data in a first data source sent by a data requesting device.

[0072] S102: Convert the format of the first data request according to the protocol standard of the first data source to obtain a second data request.

[0073] S103: Send a second data request to the first data source, so that the first data source returns first metadata of the data to be processed according to the second data request.

[0074] S104: Sending a data response including the first metadata to the data requesting device based on the Hive metadata interface service according to the protocol standard supported by the data requesting device.

[0075] The universal Hive metadata interface service can match the data request interfaces of different data request devices. Therefore, based on the metadata management method of the embodiment of the present application, there is no need to configure a corresponding metadata management device for each data request device, which can reduce the cost of metadata management.

[0076] In addition, it is possible to convert data requests to obtain data requests that match the data source, and thus it is possible to support different types of data sources. Therefore, it is also possible to manage the metadata of heterogeneous data sources and improve the efficiency of metadata management.

[0077] For step S101, the general Hive metadata interface service may be Hive Meta Store Service (metadata storage service). The data requesting device may be connected to Hive Meta Store Service, and the electronic device may be connected to the data source through Hive Meta Store Service to obtain metadata. That is, the electronic device may be connected to multiple data requesting devices through Hive Meta Store Service, and the multiple data requesting devices do not need to know the connection information of the data source, but only need to be connected to Hive Meta Store Service.

[0078] A data computing engine may be deployed in the data request device, for example, it may include: Hive, Impala, Presto, Spark, Atlas and other data computing engines. The data computing engine is used to obtain the data to be processed from the data source and process the data to be processed according to the preset data processing strategy. Before obtaining the data to be processed, the data computing engine may obtain metadata of the data to be processed from the electronic device.

[0079] The Hive metadata interface service can match different data operation engines, that is, through the Hive metadata interface service, the electronic device can connect to different data operation engines to obtain their data requests.

[0080] For step S102 and step S103, the electronic device can be connected to different data sources. The first data source can be any data source to which it is connected. For example, the first data source can be a PostGreSQL (Post Gre Structured Query Language, a free software object-relational database management system) data source, an Elastic Search (ES, a distributed, highly scalable, and highly real-time search and data analysis engine) data source, a Kafka (an open source stream processing platform, a high-throughput distributed publish-subscribe messaging system that can process all action stream data of consumers on a website) data source, or an Hbase (a distributed, column-oriented open source database) data source.

[0081] Different data sources support different protocol standards. After receiving a first data request, the electronic device can determine the data source (i.e., the first data source) to which the data requested by the first data request (i.e., the data to be processed) belongs, and convert the first data request according to the protocol standard of the first data source, so that the first data source can parse the converted first data request (i.e., the second data request).

[0082] For example, for each data source, the electronic device may locally record a conversion rule corresponding to the data source, that is, based on the conversion rule, the electronic device may convert a first data request into a second data request supported by the data source.

[0083] After receiving the second data request, the first data source may parse the second data request, obtain metadata of the data to be processed, and send the metadata to the electronic device.

[0084] With respect to step S104, after obtaining the first metadata, the electronic device can also generate a data response including the first metadata that matches the protocol standard supported by the data requesting device, and send it to the data requesting device. Furthermore, the data requesting device can parse the data response to obtain the first metadata, and obtain the data to be processed based on the first metadata to process the data to be processed.

[0085] In one embodiment, the first data source connected to the electronic device has complete metadata, that is, the database of this type of data source records the complete metadata of the stored data, for example, a PostGreSQL data source, an Elastic Search data source, and the like.

[0086] Correspondingly, based on the metadata management method provided in the embodiment of the present application, the electronic device obtains metadata (ie, first metadata) from the first data source as complete metadata.

[0087] In one embodiment, the first data source represents a structured data source; the first metadata represents a data structure of the data to be processed. Figure 2 ,exist Figure 1 On the basis of, before the above step S104, the method may further include the following steps:

[0088] S105: According to the first data request, second metadata of the attributes of the data to be processed recorded locally is obtained.

[0089] Furthermore, S104 may include the following steps:

[0090] S1041: Sending a data response including the first metadata and the second metadata to the data requesting device based on the Hive metadata interface service in accordance with the protocol standard supported by the data requesting device.

[0091] In an embodiment of the present application, if the first data source represents a structured data source, and the complete metadata of the data is not recorded in the database of this type of data source. For example, Kafka data source, Hbase data source, etc. The metadata recorded in this type of data source (i.e., the first metadata) represents the data structure of the data to be processed (for example, a table contains three columns, representing age, occupation, and education level, respectively). However, the attributes of the data to be processed (i.e., the second metadata) are not recorded in the database of this type of data source. For example, the first metadata does not record the field type of the data in each column of the table.

[0092] For this type of data source, the electronic device can obtain the pre-generated metadata (i.e., the second metadata) of the stored data except the first metadata, and store it locally. Then, when receiving the first data request, the electronic device can obtain the first metadata from the corresponding database and obtain the corresponding second metadata locally, and then the electronic device can send the complete metadata to the data requesting device.

[0093] Based on the above processing, the electronic device can be connected to a data source with complete metadata and a data source without complete metadata, that is, the electronic device can manage metadata of heterogeneous data sources at the same time, thereby improving metadata management efficiency.

[0094] In one embodiment, the electronic device can also be connected to a user-defined data source, and the metadata of this type of data source can be stored in an underlying database, or can also be stored locally in the electronic device. Accordingly, when the data requesting device requests to obtain the corresponding metadata, the electronic device can obtain the metadata from the database, and can also send the complete metadata to the data requesting device in combination with the metadata recorded locally.

[0095] For example, the electronic device may provide a metadata Service Provider Interface, through which the electronic device may connect to a custom data source.

[0096] Based on the above processing, the metadata of heterogeneous data sources can be managed in a unified manner, data silos can be reduced, the connection between data can be strengthened, and users can view metadata intuitively through electronic devices, which is conducive to discovering the value of metadata.

[0097] In one embodiment, the first data request carries the authentication information of the data requesting device. Figure 3 ,exist Figure 1 On the basis of, before the above step S103, the method may further include the following steps:

[0098] S106: Based on the information to be authenticated, an authentication request is sent to a designated authentication service.

[0099] Accordingly, the above step S103 may include the following steps:

[0100] S1031: When receiving an authentication success message returned by the authentication service, a second data request is sent to the first data source, so that the first data source returns first metadata of the data to be processed according to the second data request.

[0101] The authentication service may be a service enabled in the authentication server, and is used to authenticate the data access rights of each data requesting device.

[0102] In an embodiment of the present application, after receiving the first data request, the electronic device may also extract the information to be authenticated carried by it. The information to be authenticated may include an identifier of the data requesting device and an identifier of the data to be processed. Accordingly, the authentication service may determine whether the data requesting device has the authority to access the data to be processed based on the above identifier. If the data requesting device has the authority to access the data to be processed, the authentication service may send an authentication pass message to the electronic device. Furthermore, the electronic device may send a second data request to the first data source so that the first data source returns the first metadata based on the second data request.

[0103] Based on the above processing, each data requesting device does not need to authenticate to the authentication service separately, that is, authentication can be performed uniformly at the electronic device, which is conducive to unified management of authentication, realizes authentication decentralization, and reduces the complexity of authentication.

[0104] In addition, if the data requesting device has the authority to access the data to be processed, the authentication service may send an authentication failure message to the electronic device, and the electronic device may discard the first data request.

[0105] In one embodiment, see Figure 4 ,exist Figure 1 On the basis of, before the above step S103, the method may further include the following steps:

[0106] S107: Obtain a configuration file corresponding to the first data source recorded in the distributed storage system.

[0107] The configuration file records the connection information of the first data source.

[0108] Accordingly, the above step S103 may include:

[0109] S1032: Send a second data request to the first data source based on the connection information, so that the first data source returns first metadata of the data to be processed according to the second data request.

[0110] In the embodiment of the present application, the electronic device can locally record the configuration files of each data source. The configuration files of the data source can include the connection information of the data source, for example, the address of the data source, the user name and password for connecting to the data source, and other information.

[0111] In one implementation, you can use the following code to configure the connection information of the ES data source.

[0112] connector.name=elasticsearch

[0113] elasticsearch.coordinate=A

[0114] elasticSearch.user=B

[0115] elasticSearch.pass=C

[0116] elasticSearch.client.connectNum=D

[0117] elasticSearch.client.connectPerRoute=E

[0118] In the above code, through "connector.name = elasticsearch", the value of the field indicating the name of the connected data source is set to elasticsearch; through "elasticsearch.coordinate = A", the value of the field indicating the address of the data source is set to A; through "elasticSearch.user = B", the value of the field indicating the user name for connecting to the data source is set to B; through "elasticSearch.pass = C", the value of the field indicating the password for connecting to the data source is set to C; through "elasticSearch.client.connectNum = D", the value of the field indicating the maximum number of client connections supported by the data source is set to D; through "elasticSearch.client.connectPerRoute = E", the value of the field indicating the maximum number of parallel received requests supported by the data source is set to E.

[0119] In one implementation, the connection information of the Hbase data source can be configured using the following code.

[0120] connector.name=hbase

[0121] hbase.zookeeper.quorum=F

[0122] hbase.zookeeper.property.clientPort=G

[0123] zookeeper.znode.parent=H

[0124] In the above code, through "connector.name = hbase", the value of the field indicating the name of the connected data source is set to hbase; through "hbase.zookeeper.quorum = F", the value of the field indicating the registered address of the data source connected to the zookeeper is set to F; through "hbase.zookeeper.pro perty.clientPort = G", the value of the field indicating the port number of the zookeeper server is set to G; through "zookeeper.znode.parent = H", the value of the field indicating the address of the data source registered in zookeeper is set to H.

[0125] In addition, the conversion rules mentioned in the above embodiment can also be recorded in the configuration file. In the embodiment of the present application, the configuration file that records the connection information and the conversion rules can be called a connector of the data source.

[0126] In one embodiment, the method may further include the following steps:

[0127] Monitor metadata access events to obtain operation data of each data operation associated with metadata access events triggered within a historical time period; send the operation data to a preset queue so that a preset data analysis device obtains the operation data from the preset queue and processes based on the operation data.

[0128] In the embodiment of the present application, the electronic device can also listen to the events of accessing metadata. For example, based on the PreEvent Listener, the operation data of the operation performed before the event occurs can be obtained, based on the Event Listener, the operation data of the operation performed during the event can be obtained, and based on the EndFunction Listener, the operation data of the operation performed after the event occurs can be obtained.

[0129] After obtaining the above-mentioned operation data, the electronic device can add it to a preset queue (for example, MQ (MessageQueue)). Correspondingly, the data analysis device can obtain the operation data from the preset queue and perform analysis, auditing and other processing.

[0130] In one embodiment, the method is applied to a metadata management node, and the method may further include the following steps: based on preset monitoring indicators, obtaining access status data of data requesting devices for data to be processed within a historical time period, and system status data of the metadata management node.

[0131] Among them, the metadata management node can be the above-mentioned electronic device. In one implementation, the metadata management node can also count the status of data request devices accessing data based on HMS (Hive MetaStore, Hive metadata storage) indicators. Such indicators can be for table and partition CRUD operations (i.e., add (Create), retrieve (Retrieve), update (Update) and delete (Delete)). The metadata management node can also obtain its own system status data based on Jmx (Java Management Extensions, Java Management Extensions, a framework for implanting management functions for applications, devices, systems, etc., which can span a series of heterogeneous operating system platforms, system architectures and network transmission protocols, and flexibly develop seamlessly integrated system, network and service management applications) indicators, such as the response time of data requests, memory occupancy, and the amount of garbage data recovered.

[0132] In one embodiment, the metadata management method of the embodiment of the present application can be applied to a server cluster, which includes multiple servers, and each server can share data stored locally in the server cluster (for example, conversion rules corresponding to a data source and metadata of the data source). Each server can execute the metadata management method provided by the implementation of the present application. Exemplarily, when the server cluster receives a data request for metadata sent by a data request device, it can select a server based on the current business load of each server, and the server processes the received data request.

[0133] See also Figure 5 , Figure 5 An architectural diagram of each service layer in an electronic device provided in an embodiment of the present application.

[0134] It includes a metadata service provider layer, which provides metadata access services through MetaStore Thrift Service (metadata storage saving service), supports the same protocol as HMS (Hive MetaStore) to be compatible with open source computing engines; a data storage layer (Hadoop Distributed File System Ware House), which is used to store configuration files. It uses a distributed storage solution to synchronize configuration files; a listening service layer (MetaStoreListener), which listens to events based on PreEvent Listener, Event Listener, and EndFunction Listener, obtains relevant operation data, and sends it to the message queue to implement the metadata audit function; the service monitoring layer obtains the overall pressure of the metadata service based on Metrics indicators, specifically, including monitoring based on Java Management Extensions indicators and Hive MetaStore indicators. Hive MetaStore indicators can be CRUD operations for tables and partitions (i.e., add (Create), retrieve (Retrieve), update (Update) and delete (Delete)); a metadata service component access layer, which provides SPI (Service Provider Interface) for users to access custom data sources to implement connection management (Connection Manager).

[0135] In addition, based on the metadata service provider layer, users can customize the monitoring of various events through the Hook mechanism, such as creating table events, modifying table events, etc., and push the monitored data to the message queue in a specific format so that other devices can consume the data in the message queue.

[0136] Figure 5 In the data source, based on the native support of HMS and the support of semi-structured data, it is possible to access data sources such as Kafka, Hbase, and relational database management system (RDBMS) (for example, Post GreStructured Query Language). RawStore defines the interface for implementing metadata operations. DataNucleus provides a standard interface to access various data sources.

[0137] Connect to the Elastic Search data source through the Elastic Search connector, connect to the Post Gre Structured Query Language data source through the Post Gre Structured Query Language connector, and connect to other user-defined data sources through other connectors to achieve multi-data source support.

[0138] Based on the same inventive concept, the present application also provides a metadata management device, see Figure 6 , Figure 6 A structural diagram of a metadata management device provided in an embodiment of the present application, the device may include:

[0139] The first data request acquisition module 601 is used to receive a first data request for metadata of to-be-processed data in a first data source sent by a data request device based on a general Hive metadata interface service;

[0140] A request conversion module 602, configured to convert the format of the first data request according to the protocol standard of the first data source to obtain a second data request;

[0141] A request sending module 603, configured to send the second data request to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request;

[0142] The response module 604 is used to send a data response including the first metadata to the data request device based on the Hive metadata interface service in accordance with the protocol standard supported by the data request device.

[0143] Optionally, the first data source represents a structured data source; the first metadata represents a data structure of the data to be processed;

[0144] The device also includes:

[0145] A metadata acquisition module, configured to acquire, according to the first data request, second metadata representing the attributes of the data to be processed and recorded locally, before sending a data response including the first metadata to the data request device based on the Hive metadata interface service in accordance with the protocol standard supported by the data request device;

[0146] The response module 604 is specifically configured to send a data response including the first metadata and the second metadata to the data request device based on the Hive metadata interface service in accordance with a protocol standard supported by the data request device.

[0147] Optionally, the first data request carries information to be authenticated of the data requesting device;

[0148] The device also includes:

[0149] an authentication request sending module, configured to send an authentication request to a designated authentication service based on the information to be authenticated before sending the second data request to the first data source so that the first data source returns the first metadata of the data to be processed according to the second data request;

[0150] The request sending module 603 is specifically configured to send the second data request to the first data source upon receiving an authentication success message returned by the authentication service, so that the first data source returns the first metadata of the data to be processed according to the second data request.

[0151] Optionally, the device further comprises:

[0152] a configuration file acquisition module, configured to acquire a configuration file corresponding to the first data source recorded in a distributed storage system before sending the second data request to the first data source so that the first data source returns the first metadata of the data to be processed according to the second data request; wherein the configuration file records the connection information of the first data source;

[0153] The request sending module 603 is specifically configured to send the second data request to the first data source based on the connection information, so that the first data source returns the first metadata of the data to be processed according to the second data request.

[0154] Optionally, the device further comprises:

[0155] A monitoring module is used to monitor metadata access events and obtain operation data of various data operations associated with metadata access events triggered within a historical time period;

[0156] The operation data sending module is used to send the operation data to a preset queue, so that a preset data analysis device obtains the operation data from the preset queue and processes the operation data based on the operation data.

[0157] Optionally, the device is applied to a metadata management node, and the device further includes:

[0158] The monitoring module is used to obtain the access status data of the data requesting device for the data to be processed and the system status data of the metadata management node within a historical time period based on preset monitoring indicators.

[0159] The universal Hive metadata interface service can match the data request interfaces of different data request devices. Therefore, based on the metadata management method of the embodiment of the present application, there is no need to configure a corresponding metadata management device for each data request device, which can reduce the cost of metadata management.

[0160] In addition, it is possible to convert data requests to obtain data requests that match the data source, and thus it is possible to support different types of data sources. Therefore, it is also possible to manage the metadata of heterogeneous data sources and improve the efficiency of metadata management.

[0161] The present application also provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0162] Memory 703, used for storing computer programs;

[0163] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0164] Based on a general Hive metadata interface service, receiving a first data request for metadata of to-be-processed data in a first data source sent by a data requesting device;

[0165] Convert the first data request into a second data request according to a protocol standard of the first data source;

[0166] Sending the second data request to the first data source, so that the first data source returns first metadata of the data to be processed according to the second data request;

[0167] According to the protocol standard supported by the data requesting device and based on the Hive metadata interface service, a data response including the first metadata is sent to the data requesting device.

[0168] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0169] The communication interface is used for communication between the above electronic device and other devices.

[0170] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] The embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the metadata management method provided by the embodiment of the present application.

[0173] The embodiment of the present application also provides another computer program product including instructions, which, when executed on a computer, enables the computer to execute the metadata management method provided by the embodiment of the present application.

[0174] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0175] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0176] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0177] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A metadata management method, characterized in that: The method comprises: Based on a general Hive metadata interface service, receiving a first data request for metadata of to-be-processed data in a first data source sent by a data requesting device; Convert the first data request into a second data request according to a protocol standard of the first data source; Sending the second data request to the first data source, so that the first data source returns first metadata of the data to be processed according to the second data request; According to a protocol standard supported by the data requesting device and based on the Hive metadata interface service, sending a data response including the first metadata to the data requesting device; The first data request carries the to-be-authenticated information of the data requesting device; Before sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request, the method further includes: Based on the information to be authenticated, sending an authentication request to a designated authentication service; The sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request includes: When receiving the authentication success message returned by the authentication service, the second data request is sent to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request.

2. The method according to claim 1, characterized in that The first data source represents a structured data source; the first metadata represents a data structure of the data to be processed; Before sending a data response including the first metadata to the data requesting device based on the Hive metadata interface service according to the protocol standard supported by the data requesting device, the method further includes: According to the first data request, obtaining second metadata recorded locally and representing attributes of the data to be processed; The sending a data response including the first metadata to the data requesting device based on the Hive metadata interface service according to the protocol standard supported by the data requesting device includes: According to the protocol standard supported by the data requesting device and based on the Hive metadata interface service, a data response including the first metadata and the second metadata is sent to the data requesting device.

3. The method according to claim 1, characterized in that Before sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request, the method further includes: Obtaining a configuration file corresponding to the first data source recorded in a distributed storage system; wherein the configuration file records connection information of the first data source; The sending the second data request to the first data source so that the first data source returns the first metadata of the to-be-processed data according to the second data request includes: Based on the connection information, the second data request is sent to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request.

4. The method according to claim 1, characterized in that: The method further comprises: Monitor metadata access events to obtain operation data of each data operation associated with metadata access events triggered within a historical time period; The operation data is sent to a preset queue, so that a preset data analysis device obtains the operation data from the preset queue and performs processing based on the operation data.

5. The method according to claim 1, characterized in that The method is applied to a metadata management node, and the method further comprises: Based on preset monitoring indicators, access status data of the data requesting device for the data to be processed and system status data of the metadata management node within a historical time period are obtained.

6. A metadata management device, characterized in that: The device comprises: A first data request acquisition module, configured to receive, based on a general Hive metadata interface service, a first data request for metadata of to-be-processed data in a first data source sent by a data request device; a request conversion module, configured to convert the format of the first data request according to a protocol standard of the first data source to obtain a second data request; a request sending module, configured to send the second data request to the first data source, so that the first data source returns the first metadata of the data to be processed according to the second data request; A response module, configured to send a data response including the first metadata to the data request device based on the Hive metadata interface service in accordance with a protocol standard supported by the data request device; The first data request carries the to-be-authenticated information of the data requesting device; The device also includes: an authentication request sending module, configured to send an authentication request to a designated authentication service based on the information to be authenticated before sending the second data request to the first data source so that the first data source returns the first metadata of the data to be processed according to the second data request; The request sending module is specifically configured to send the second data request to the first data source upon receiving an authentication success message returned by the authentication service, so that the first data source returns the first metadata of the data to be processed according to the second data request.

7. The device according to claim 6, characterized in that The first data source represents a structured data source; the first metadata represents a data structure of the data to be processed; The device also includes: A metadata acquisition module, configured to acquire, according to the first data request, second metadata representing the attributes of the data to be processed recorded locally before sending a data response including the first metadata to the data request device based on the Hive metadata interface service in accordance with the protocol standard supported by the data request device; The response module is specifically configured to send a data response including the first metadata and the second metadata to the data request device based on the Hive metadata interface service in accordance with a protocol standard supported by the data request device.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 5 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 5 are implemented.

Images