Metadata processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a metadata processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: extracting a system table of metadata of each database from a distributed database cluster; for a system table of any database, obtaining keys in a key value pair from a database identifier of the database, a system table identifier of the system table, an attribute identifier of a business data attribute of metadata and a business data identifier to which an attribute value corresponding to the attribute identifier belongs; taking the attribute value as a value in the key value pair to obtain an initial key value pair corresponding to each attribute value of each business data attribute; compressing the initial key value pair to obtain a target key value pair corresponding to each attribute value of each business data attribute; and writing the target key value pair into a metadata storage database corresponding to the distributed database cluster. By adopting the method, the metadata storage space can be saved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a metadata processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of distributed database technology, enterprises' demand for data continues to increase, and various data objects are constantly imported from isolated application systems into distributed databases for storage. Faced with the massive and continuously increasing variety of data objects in distributed databases, enterprises need to understand the source of data and its changes over time.

[0003] Currently, metadata is mainly used to trace the upstream and downstream links of data. Metadata, also known as intermediary data or relay data, is data that describes data, mainly information that describes data attributes, and is used to support functions such as indicating storage locations, historical data, resource searches, and file records. Metadata is usually stored in the corresponding database, but as the amount of metadata increases, the space occupied by metadata also increases, resulting in insufficient storage space in the database for the required business data. Summary of the invention

[0004] Based on this, it is necessary to provide a metadata processing method, apparatus, computer device, computer-readable storage medium and computer program product that can save metadata storage space in response to the above technical problems.

[0005] In a first aspect, the present application provides a metadata processing method, comprising:

[0006] Extract the system table corresponding to each database's metadata from the distributed database cluster;

[0007] For the system table corresponding to any database, obtain the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute;

[0008] Compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of each business data attribute;

[0009] Write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0010] In a second aspect, the present application further provides a metadata processing device, comprising:

[0011] A system table generation module is used to extract the system tables of metadata corresponding to each database from the distributed database cluster;

[0012] A key-value pair generation module is used to obtain the key in the key-value pair from the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, for the system table corresponding to any database, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute;

[0013] A compression module is used to compress the initial key-value pairs to obtain target key-value pairs corresponding to each attribute value of each business data attribute;

[0014] The storage module is used to write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0015] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Extract the system table corresponding to each database's metadata from the distributed database cluster;

[0017] For the system table corresponding to any database, obtain the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute;

[0018] Compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of each business data attribute;

[0019] Write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0020] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0021] Extract the system table corresponding to each database's metadata from the distributed database cluster;

[0022] For the system table corresponding to any database, obtain the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute;

[0023] Compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of each business data attribute;

[0024] Write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0025] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0026] Extract the system table corresponding to each database's metadata from the distributed database cluster;

[0027] For the system table corresponding to any database, obtain the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute;

[0028] Compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of each business data attribute;

[0029] Write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0030] The metadata processing method, apparatus, computer equipment, computer-readable storage medium and computer program product can convert the data content of the system table into key-value pairs by extracting the system table's database identifier, system table identifier, attribute identifier and business data identifier, and the attribute values ​​corresponding to the attribute identifier and the business data identifier from the distributed database cluster, and then efficiently read the metadata; then compress each initial key-value pair to obtain a target key-value pair, and store each target key-value pair in a metadata storage database, thereby saving the storage space of the target key-value pair and thus saving the storage space of the metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 A diagram of an application environment of a metadata processing method in an embodiment;

[0033] Figure 2 A schematic diagram of a metadata processing method in one embodiment;

[0034] Figure 3 A schematic diagram of a tree structure encoding value in one embodiment;

[0035] Figure 4 is a schematic diagram of a distributed database cluster in one embodiment;

[0036] Figure 5 A schematic diagram of a process of metadata reading in one embodiment;

[0037] Figure 6 A schematic diagram of a process of writing metadata in one embodiment;

[0038] Figure 7 is a structural block diagram of a metadata processing device in one embodiment;

[0039] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0040] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The metadata processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the distributed database cluster 104 through the network, and the distributed database cluster 104 can receive and store the business data sent by the terminal 102. The distributed database cluster 104 includes multiple database servers. In one embodiment, the distributed database cluster 104 extracts the system table of metadata corresponding to each database; for the system table corresponding to any database, the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, obtain the key in the key-value pair, and use the attribute value as the value in the key-value pair to obtain the initial key-value pair corresponding to each attribute value of each business data attribute; compress the initial key-value pair to obtain the target key-value pair corresponding to each attribute value of each business data attribute; write the target key-value pair into the metadata storage database corresponding to the distributed database cluster. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head mounted device, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0043] In an exemplary embodiment, Figure 2 As shown, a metadata processing method is provided, which is applied to Figure 1 The distributed database cluster in is used as an example to illustrate the following:

[0044] Step 202: extracting the system tables of metadata corresponding to each database from the distributed database cluster.

[0045] Among them, the distributed database cluster refers to the cluster of distributed database servers, including various database servers, which are used to store business data. Metadata refers to data that describes data, which can characterize the data characteristics of business data stored in the database, such as data content, data structure, data size and other data characteristics of business data, and is also used to support functions such as indicating storage location, historical data, resource search, file records, etc. System tables are a set of tables inside the distributed database cluster for storing and managing database metadata, which record information about database structure, user permissions, indexes, views and other database objects.

[0046] Specifically, the distributed database cluster includes various database servers, which can be referred to as databases for short. Each database stores business data sent in advance by the terminal, and the business data is stored in the database in the form of a table as a business data table. Metadata is generated in each database according to the attribute characteristics of the stored business data table, such as the name, source, type, etc. of the business data table. A database server dedicated to storing metadata is set in the distributed database cluster as a metadata storage database. The distributed database cluster obtains metadata corresponding to each database from each database, and converts the metadata into system tables corresponding to each database according to a preset format, that is, the metadata exists in the form of system tables in each database, and generates system table identifiers corresponding to each system table, and associates each system table identifier with the corresponding database identifier.

[0047] In one embodiment, a blank initial system table is pre-set in each database. After new business data is written, the initial system table is updated according to the information of the existing business data to obtain an updated system table corresponding to the database. The updated system table corresponding to each database is extracted from the distributed database cluster as the system table of metadata corresponding to each database.

[0048] Step 204, for the system table corresponding to any database, obtain the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute.

[0049] Among them, business data attributes refer to the attribute characteristics that characterize the business data table. Business data identifier refers to a specific column or combination of multiple columns in the business data table generated in the database, which is used to uniquely identify the data content in the table and can be the primary key corresponding to the business data table in the database.

[0050] Specifically, each database extracts from the system table the database identifier of the database to which the system table belongs, the system table identifier of the system table, the attribute identifier of each business data attribute corresponding to the metadata, and the business data identifier to which each attribute value corresponding to the attribute identifier belongs. The attribute identifier of each business data attribute is, for example, table_name (the table name of the business data table), table_space (the space occupied by the business data table), etc. The attribute value corresponding to the attribute identifier table_name is, for example, name1, name2, etc. The attribute values ​​of the same row in the system table all correspond to the business data identifier of a business data table.

[0051] For the system table corresponding to any database, the database identifier, system table identifier, attribute identifier and business data identifier are concatenated to obtain a concatenated string. The concatenated string is used as the key in the key-value pair. Then, the attribute value corresponding to the attribute identifier and the business data identifier is used as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute in the system table.

[0052] Step 206: compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of the business data attributes.

[0053] Step 208: Write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0054] Specifically, extract the public fields and non-public fields from the key in the initial key-value pair, encode the public fields and non-public fields to obtain the encoded value, and perform string compression on the value in the initial key-value pair to obtain the compressed value. According to the encoded value and the compressed value, obtain the target key-value pairs corresponding to each attribute value of each business data attribute in the system table, and then store each target key-value pair corresponding to each system table in the metadata storage database corresponding to the distributed database cluster.

[0055] In the above metadata processing method, by extracting the system table of metadata corresponding to each database from the distributed database cluster; extracting the database identifier, system table identifier, attribute identifier and business data identifier of the system table, and the attribute values ​​corresponding to the attribute identifier and the business data identifier to generate each initial key-value pair corresponding to the system table, the data content of the system table can be converted into each key-value pair, and the key-value pair can be used to efficiently read the metadata; then, each initial key-value pair is compressed to obtain a target key-value pair, and each target key-value pair is stored in a metadata storage database, which saves the storage space of the target key-value pair, thereby saving the storage space of the metadata.

[0056] In one embodiment, step 206, compressing the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of each business data attribute, includes:

[0057] For an initial key-value pair corresponding to the attribute value of any business data attribute, based on the key in the initial key-value pair, extract the common fields and the non-common fields from the fields corresponding to the database identifier, the system table identifier, the attribute identifier and the business data identifier respectively;

[0058] Mark the node fields in the public fields and the non-public fields, and build a tree structure encoding value based on the public fields, the non-public fields, the marked node fields and the field order corresponding to each field, so as to obtain the key in the target key-value pair corresponding to each attribute value of each business data attribute;

[0059] The values ​​in the initial key-value pair are encoded to obtain the values ​​in the target key-value pair corresponding to each attribute value of each business data attribute.

[0060] Among them, public fields refer to repeated fields in database identifiers, system table identifiers, and attribute identifiers. Non-public fields refer to the remaining non-public fields in database identifiers, system table identifiers, and attribute identifiers. Node fields refer to marker fields that can be used as end fields. Tree structure coded values ​​refer to data in the form of a tree structure, which represent the coding results of database identifiers, system table identifiers, and attribute identifiers.

[0061] Specifically, for the initial key-value pair corresponding to the attribute value of any business data attribute, the key in the initial key-value pair is a concatenated string obtained by concatenating the fields corresponding to the database identifier, the system table identifier, and the attribute identifier. Extract the common fields from each string in the concatenated string, and use the remaining fields as non-public fields. Mark the node fields in the public fields and non-public fields, and generate a tree structure coding value for the public nodes, non-public nodes, and marked nodes according to the field order corresponding to each field. Then, the business data identifier and the tree structure coding value are concatenated, and the keys in the target key-value pairs corresponding to each attribute value of each business data attribute are obtained according to the concatenated coding value, such as Figure 3 The schematic diagram of the tree structure encoding value shown in the figure, the branch of each node of the tree corresponds to a field, such as the field p corresponding to node 1, and the branch links composed of branches correspond to a string, such as pa, pen, zero, zen, axe, at and other strings. The field and node sequence number corresponding to each branch can be determined according to the field order of each field in the string; node 0 is the root node, nodes 4, 9, 13, 11, and 12 are all marking nodes, indicating the last field corresponding to each string, and the field of the previous node corresponding to the marking node is a public field, such as nodes 1, 2, 3, and 6 correspond to fields p, z, a, and e, respectively, and the fields of the remaining node pairs are non-public fields.

[0062] Among them, the system table schema name (database identifier) ​​occupies 1 byte of storage, the table name (system table identifier) ​​occupies 2 bytes of storage, the column name (attribute identifier) ​​occupies 2 bytes of storage, and the primary key value (business data identifier) ​​occupies 4 bytes or 8 bytes of space.

[0063] Then, the value in the initial key-value pair is encoded according to a preset encoding algorithm to obtain the value in the target key-value pair corresponding to each attribute value of each business data attribute.

[0064] In this embodiment, by encoding the key in the initial key-value pair into a tree structure coded value and compressing the value in the initial key-value pair, the field compression of the initial key-value pair is achieved, saving the storage space of the target key-value pair. Furthermore, the tree structure coded value is constructed according to the database identifier, the system table identifier, and the attribute identifier, without the need to involve the easily changeable business data identifier in the construction of the tree structure coded value, so that the accuracy of the tree structure coded value can be guaranteed. In one embodiment, the distributed database cluster includes a cache service layer; the cache service layer includes metadata caches corresponding to each database in the distributed database cluster; step 208, writing the target key-value pair into the metadata storage database corresponding to the distributed database cluster, includes:

[0065] Send each target key-value pair corresponding to each database to the corresponding metadata cache in the cache service layer;

[0066] For any metadata cache, each target key-value pair is sent to the metadata storage database for storage, and high-frequency key-value pairs in each target key-value pair are identified and stored in the metadata cache;

[0067] Through metadata caching, high-frequency key-value pairs are synchronously updated to the database caches corresponding to other databases in the cache service layer.

[0068] The cache service layer is used to provide cache services for each database in the distributed database cluster. Metadata cache refers to the cache service that stores metadata for each database in the cache service layer. High-frequency key-value pairs refer to key-value pairs with a high storage frequency.

[0069] Specifically, after obtaining the target key-value pairs of the system tables corresponding to each database, each target key-value pair is sent to the corresponding metadata cache in the cache service layer, and each target key-value pair is sent to the metadata storage database for storage through each metadata cache. Then each metadata cache identifies high-frequency key-value pairs in each target key-value pair, and the high-frequency key-value pairs can be determined according to the type of the target key-value pair, a preset high-frequency tag, etc. Then each metadata cache stores the high-frequency key-value pairs, and synchronizes and updates the high-frequency key-value pairs to the metadata caches corresponding to other databases in the cache service layer.

[0070] In this embodiment, each target key-value pair is sent to the metadata storage database through the cache service layer, and high-frequency key-value pairs are stored so that the existing target key-value pairs can be quickly read from the cache service layer later, thereby improving the metadata reading efficiency.

[0071] In one embodiment, in step 208, after writing the target key-value pair into the metadata storage database corresponding to the distributed database cluster, the metadata processing method further includes:

[0072] In response to a metadata read request for a current key-value pair in a target database, generating a query key based on metadata query information carried in the metadata read request;

[0073] Sending a query key to a database cache corresponding to a target database in a cache service layer, and performing character matching of a key-value pair based on the query key in the database cache corresponding to the target database;

[0074] When the current key-value pair is matched in the database cache corresponding to the target database, the current key-value pair in the database cache corresponding to the target database is sent to the request end corresponding to the metadata read request.

[0075] Specifically, the request terminal sends a metadata read request for the current key-value pair in the target database to the distributed database cluster. According to the metadata query information carried by the metadata read request, the metadata query information may be at least one of the database identifier, system table identifier, attribute identifier, and business data identifier to be queried. A tree structure encoding value is generated according to the metadata query information, and the tree structure encoding value is used as the query key. The query key is sent to the database cache corresponding to the target database in the cache service layer, and the query key is matched in each stored key-value pair in the database cache corresponding to the target database. When there is a current key-value pair matching the query key in the database cache corresponding to the target database, the current key-value pair is decoded and the decoded current key-value pair is sent to the request terminal corresponding to the metadata read request. For example, if the metadata query information is: database identifier A-system table identifier a, the current key-value pair matched according to the query key corresponding to the metadata query information is represented as each business data table (name1, name2, name3, ...) recorded in the system table corresponding to the system table identifier a in the database corresponding to the database identifier A. Then, the request terminal obtains the business data corresponding to the business data table name1 from the target database according to the value in the returned current key-value pair, such as the business data table name1.

[0076] In a specific embodiment, a prefix search can be performed in the database cache corresponding to the target database according to the query key, and the common field in the tree structure encoding value can be used as the prefix field, such as Figure 3 Here, ze is the prefix field. Field matching is performed in the tree structure coded value according to the field in the query key. When the field corresponding to the query key matches the prefix field of the tree structure coded value, all strings containing the prefix field in the tree structure coded value are output, such as zero and zen strings.

[0077] In this embodiment, by sending a query key to the database cache corresponding to the target database in the cache service layer, key-value pair matching is performed based on the query key in the database cache corresponding to the target database, thereby improving the efficiency of metadata reading.

[0078] In one embodiment, the metadata processing method further includes:

[0079] When the target metadata corresponding to the query request does not exist in the database cache corresponding to the target database, the target metadata is read from the metadata storage database through the cache service layer and stored in the database cache corresponding to the target database;

[0080] The target metadata is synchronously updated to the database caches corresponding to other databases in the distributed database cluster through the cache service layer.

[0081] Specifically, when the current key-value pair corresponding to the query key is not matched in the database cache corresponding to the target data, the cache service layer requests the current key-value pair corresponding to the query key from the metadata storage database, decodes the current key-value pair, restores the tree structure encoding value to the corresponding character, and returns the decoded current key-value pair to the request end corresponding to the metadata read request. Then the database cache corresponding to the target database stores the undecoded current key-value pair, and synchronizes and updates the database caches corresponding to other databases in the cache service layer according to the current key-value pair.

[0082] In this embodiment, by synchronously updating the current key-value pair to the database cache corresponding to other databases, the requesting end can obtain the same current key-value pair in the database cache corresponding to other databases, thereby improving the reading efficiency of metadata.

[0083] In one embodiment, the distributed database cluster includes a request service layer connected to the cache service layer; before extracting metadata corresponding to each database from the distributed database cluster in step 202, the metadata processing method further includes:

[0084] Obtain the cache socket library of each database in the distributed database cluster, and create cache sockets corresponding to each database in the cache service layer;

[0085] Obtain the request socket library corresponding to the request service layer, and generate request sockets corresponding to each database in the request service layer;

[0086] The cache sockets corresponding to each database are respectively bound to the communication information of the corresponding database, and the cache sockets corresponding to each database are set to listening mode. When the cache service layer receives a connection request sent by the request service layer, the current cache socket corresponding to the current request socket is sent to the request service layer according to the current request socket in the connection request, so that the request service layer communicates with the cache service layer through the communication information bound to the current cache socket and receives the metadata reading request.

[0087] Among them, the request service layer is used to execute the processing of metadata read requests or metadata write requests for the distributed database cluster. The request socket and the cache socket are used to provide a communication interface between the request service layer and the cache service layer. The socket refers to the communication protocol pre-defined in the request service layer and the cache service layer, and the corresponding communication interface can be provided for the request service layer and the cache service layer according to the communication protocol corresponding to the socket. The request socket library refers to the collection of sockets set for the request service layer, and the sockets in the request socket library are used as request sockets. The cache socket library refers to the collection of sockets set for the cache service layer, and the sockets in the cache socket library are used as cache sockets. The current request socket refers to the socket currently selected by the request service layer for communication with the cache service layer. The current cache socket refers to the cache socket corresponding to the cache service layer that can match the current request socket for communication. Specifically, obtain the cache socket library of each database in the distributed database cluster, create cache sockets corresponding to each database in the cache service layer, bind the cache sockets to the communication information corresponding to each database, such as the local address and IP address of the database, and set the cache sockets corresponding to each database to the listening mode. Obtain the request socket library corresponding to the request service layer, and generate request sockets corresponding to each database in the request service layer. When the cache service layer receives the connection request sent by the request service layer, according to the current request socket in the connection request, the current cache socket corresponding to the current request socket is sent to the request service layer. When the match is successful, the request service layer communicates with the cache service layer through the communication information bound to the current cache socket, and can receive metadata read requests or metadata write requests through the request service layer.

[0088] In this embodiment, the communication between the request service layer and the cache service layer is implemented through sockets, thereby improving the communication efficiency.

[0089] In one of the examples, the request service layer receives at least one metadata read request, responds to the metadata read request using event-driven method, and asynchronously processes each metadata read request through asynchronous I / O.

[0090] In a specific embodiment, Figure 4As shown, a schematic diagram of a distributed database cluster is provided. The distributed data cluster includes a computing layer, a stateless service layer (request service layer), a metadata cache layer (cache service layer) and a metadata storage layer (metadata storage database). Among them, the stateless service layer, the metadata cache layer and the metadata storage layer serve as the metadata layer.

[0091] The computing layer is used to generate key-value pairs for metadata and encode and compress the key-value pairs;

[0092] A stateless service layer that responds to requests for metadata;

[0093] Metadata cache layer, used to cache metadata;

[0094] The metadata storage layer includes various KV (key-value) databases for storing metadata.

[0095] In a specific embodiment, Figure 5 As shown in the figure, a flowchart of metadata reading is provided. When reading the system table, the computing layer encodes the metadata acquisition information into a key according to the SQL information, and initiates a query request to the stateless service layer. The stateless service layer processes the request in an event-driven and asynchronous I / O manner, and then initiates a read request to the local metadata cache layer through a UNIX socket. If the metadata cache layer finds that there is already a local cache, the data is returned directly; otherwise, the key is used to request the value from the metadata storage layer. After obtaining the data from the metadata storage layer, the cache is established according to the LRU algorithm, and the caches of other machines in the cluster are updated synchronously. After the stateless service layer returns the data to the computing layer, the computing layer decompresses it using a decompression algorithm.

[0096] In a specific embodiment, Figure 6 As shown in the figure, a flowchart of metadata writing is provided. When writing to the system table, the computing layer first encodes and compresses the system table into Key and Value according to the SQL information, and then initiates a request to the stateless service layer. The stateless service layer initiates a write request to the local metadata cache layer through the UNIX socket. While updating the local cache, the metadata cache layer notifies the metadata cache layers on other hosts in the metadata cluster to update the cache, and then writes the data to the metadata storage layer. The stateless service layer returns a write success notification to the computing layer.

[0097] In this embodiment, by splitting metadata from each database of the distributed database cluster for separate storage, horizontal expansion of the computing layer can be achieved, thereby improving the processing efficiency of the database in coping with massive concurrent requests.

[0098] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a metadata processing device for implementing the metadata processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more metadata processing device embodiments provided below can refer to the limitations of the metadata processing method above, and will not be repeated here.

[0100] In an exemplary embodiment, Figure 7 As shown, a metadata processing device 700 is provided, including: a system table generation module 702, a key-value pair generation module 704, a compression module 706 and a storage module 708, wherein:

[0101] A system table generation module 702 is used to extract the system table of metadata corresponding to each database from the distributed database cluster;

[0102] The key-value pair generating module 704 is used for obtaining the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, for the system table corresponding to any database, and using the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute;

[0103] A compression module 706 is used to compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of each business data attribute;

[0104] The storage module 708 is used to write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

[0105] In one embodiment, the compression module 706 is also used to extract common fields and non-common fields from the fields corresponding to the database identifier, the system table identifier, the attribute identifier and the business data identifier, respectively, based on the key in the initial key-value pair corresponding to the attribute value of any business data attribute; mark the node fields in the common fields and the non-common fields, and build a tree structure encoding value based on the common fields, the non-common fields, the marked node fields and the field order corresponding to each field, to obtain the keys in the target key-value pairs corresponding to each attribute value of each business data attribute; encode the values ​​in the initial key-value pairs to obtain the values ​​in the target key-value pairs corresponding to each attribute value of each business data attribute.

[0106] In one embodiment, the distributed database cluster includes a cache service layer; the cache service layer includes metadata caches corresponding to each database in the distributed database cluster; the storage module 708 is also used to send each target key-value pair corresponding to each database to the corresponding metadata cache in the cache service layer; for any metadata cache, each target key-value pair is sent to the metadata storage database for storage, and high-frequency key-value pairs in each target key-value pair are identified, and the high-frequency key-value pairs are stored in the metadata cache; through the metadata cache, the high-frequency key-value pairs are synchronously updated to the metadata caches corresponding to other databases in the cache service layer.

[0107] In one embodiment, the metadata processing device 700 is also used to respond to a metadata read request for a current key-value pair in a target database, generate a query key based on metadata query information carried in the metadata read request; send the query key to a database cache corresponding to the target database in the cache service layer, and perform character matching of the key-value pair based on the query key in the database cache corresponding to the target database; when the current key-value pair is matched in the database cache corresponding to the target database, send the current key-value pair in the database cache corresponding to the target database to the request end corresponding to the metadata read request.

[0108] In one embodiment, the metadata processing device 700 is also used to read the current metadata from the metadata storage database through the cache service layer and store it in the database cache corresponding to the target database when the current key-value pair is not matched in the database cache corresponding to the target database; and synchronously update the current metadata to the database caches corresponding to other databases in the distributed database cluster through the cache service layer.

[0109] In one embodiment, the distributed database cluster includes a request service layer connected to the cache service layer; the metadata processing device 700 is also used to obtain the cache socket library of each database in the distributed database cluster, and create a cache socket corresponding to each database in the cache service layer; obtain the request socket library corresponding to the request service layer, and generate a request socket corresponding to each database in the request service layer; bind the cache socket corresponding to each database to the communication information of the corresponding database, and set the cache socket corresponding to each database to the listening mode, so that when the cache service layer receives the connection request sent by the request service layer, according to the current request socket in the connection request, the current cache socket corresponding to the current request socket is sent to the request service layer, so that the request service layer communicates with the cache service layer through the communication information bound to the current cache socket and receives the metadata reading request.

[0110] Each module in the metadata processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0111] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store business data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a metadata processing method is implemented.

[0112] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a metadata processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0113] Those skilled in the art will understand that Figure 8-Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0114] In an exemplary embodiment, in one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0116] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0118] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0119] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0120] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A metadata processing method, characterized in that: Applied to a distributed database cluster; the method comprises: Extracting a system table of metadata corresponding to each database from the distributed database cluster; For a system table corresponding to any database, obtain the key in the key-value pair by using the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, and use the attribute value as the value in the key-value pair to obtain initial key-value pairs corresponding to each attribute value of each business data attribute; Compressing the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of the business data attributes; The target key-value pair is written into the metadata storage database corresponding to the distributed database cluster.

2. The method according to claim 1, characterized in that The compressing the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of the business data attributes respectively includes: For an initial key-value pair corresponding to the attribute value of any business data attribute, based on the key in the initial key-value pair, extracting common fields and non-common fields from the fields corresponding to the database identifier, the system table identifier, the attribute identifier and the business data identifier respectively; Marking node fields in the public fields and the non-public fields, constructing a tree structure encoding value based on the public fields, the non-public fields, the marked node fields, and the field order corresponding to each field, and obtaining the keys in the target key-value pairs corresponding to each attribute value of each business data attribute; The values ​​in the initial key-value pairs are encoded to obtain the values ​​in the target key-value pairs corresponding to the attribute values ​​of the business data attributes.

3. The method according to claim 1, characterized in that: The distributed database cluster includes a cache service layer; the cache service layer includes metadata caches corresponding to each database in the distributed database cluster; Writing the target key-value pair into the metadata storage database corresponding to the distributed database cluster includes: Sending each target key-value pair corresponding to each database to the corresponding metadata cache in the cache service layer; For any metadata cache, each target key-value pair is sent to a metadata storage database for storage, and a high-frequency key-value pair among each target key-value pair is identified, and the high-frequency key-value pair is stored in the metadata cache; Through the metadata cache, the high-frequency key-value pairs are synchronously updated to the metadata caches corresponding to other databases in the cache service layer.

4. The method according to claim 3, characterized in that After writing the target key-value pair into the metadata storage database corresponding to the distributed database cluster, the method further includes: In response to a metadata read request for a current key-value pair in a target database, generating a query key based on metadata query information carried in the metadata read request; Sending the query key to a database cache corresponding to the target database in the cache service layer, and performing character matching of key-value pairs based on the query key in the database cache corresponding to the target database; When the current key-value pair is matched in the database cache corresponding to the target database, the current key-value pair in the database cache corresponding to the target database is sent to the request end corresponding to the metadata read request.

5. The method according to claim 4, characterized in that The method further comprises: When the current key-value pair is not matched in the database cache corresponding to the target database, the current metadata is read from the metadata storage database through the cache service layer and stored in the database cache corresponding to the target database; The current metadata is synchronously updated to database caches corresponding to other databases in the distributed database cluster through the cache service layer.

6. The method according to claim 1, characterized in that The distributed database cluster includes a request service layer connected to a cache service layer; before extracting metadata corresponding to each database from the distributed database cluster, the method further includes: Obtaining a cache socket library of each database in the distributed database cluster, and creating cache sockets corresponding to each database in the cache service layer; Obtaining a request socket library corresponding to the request service layer, and generating request sockets corresponding to each of the databases respectively in the request service layer; The cache sockets corresponding to the various databases are respectively bound to the communication information of the corresponding databases, and the cache sockets corresponding to the various databases are set to the listening mode, so that when the cache service layer receives the connection request sent by the request service layer, the current cache socket corresponding to the current request socket is sent to the request service layer according to the current request socket in the connection request, so that the request service layer communicates with the cache service layer through the communication information bound to the current cache socket and receives the metadata reading request.

7. A metadata processing device, characterized in that: The device comprises: A system table generation module is used to extract the system tables of metadata corresponding to each database from the distributed database cluster; A key-value pair generation module is used to obtain the key in the key-value pair from the database identifier corresponding to the database, the system table identifier corresponding to the system table, the attribute identifier of the business data attribute corresponding to the metadata, and the business data identifier to which the attribute value corresponding to the attribute identifier belongs, for the system table corresponding to any database, and use the attribute value as the value in the key-value pair to obtain the initial key-value pairs corresponding to each attribute value of each business data attribute; A compression module, used to compress the initial key-value pairs to obtain target key-value pairs corresponding to the attribute values ​​of the business data attributes; The storage module is used to write the target key-value pair into the metadata storage database corresponding to the distributed database cluster.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images