Distributed database supports massive version metadata storage method and retrieval method

Abstract

The present invention relates to the field of cloud storage technology, and discloses a distributed database supporting a massive version of metadata storage method and a retrieval method, including: obtaining initial metadata and a timestamp corresponding to the initial metadata; adding the timestamp as version information to the initial key and modifying the initial key; sorting the modified metadata from large to small by the timestamp to construct a large-capacity table. The present invention has the advantage of wide applicability and is applicable to most distributed key-value databases, as long as the database has the ability to sort according to the lexicographic order of the keys. Without affecting the function and performance of the distributed key-value database itself, it is achieved that there is no upper limit on the number of versions of metadata records supported. The retrieval performance and service quality are not affected by the number of versions of the metadata record, which solves the problems of load imbalance, performance degradation, service unavailability, etc. faced by large-capacity tables in distributed key-value databases when storing massive versions of metadata records.

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a distributed database supporting a massive version metadata storage method and a retrieval method. Background Art

[0002] In a massive data storage system, a large-capacity table is usually used to store metadata information of all data. The typical characteristics of these data are that the data type is mainly unstructured or semi-structured, massive (EB level), and each data file or object may have several versions. This requires that this large-capacity table, in addition to being highly scalable, highly available, and highly reliable, also needs to support massive versions for metadata records with the same primary key and ensure stable performance, that is, the read and write performance of metadata records will not fluctuate significantly due to the sharp increase in the number of metadata records with different primary keys in the table, nor will it fluctuate significantly due to the sharp increase in the number of different versions of metadata records with the same primary key in the table. Among them, a large-capacity table refers to a table that stores massive metadata records in a distributed key-value database.

[0003] In order to meet the above requirements, a distributed key-value database is generally used to store this large-capacity table in related technologies. This is a non-relational database, and each metadata record is a key-value pair.<key,value> , the key is unique, and the dictionary is sorted with the key as the index. As the table capacity grows, the entire table will be divided into several partitions according to the key. Each partition contains a continuous key-value pair. When the capacity of a partition exceeds a certain threshold, the partition will be split into two sub-partitions, which are evenly distributed on different servers of the database. Distributed key-value databases better meet the various requirements of massive data storage systems for metadata storage. However, when some keys have massive versions, all versions of a key will be concentrated in a partition and cannot be divided. This will cause a partition to be extremely large, which will not only destroy the load balancing of the database, but in extreme cases, it will also cause the database servers where these partitions are located to be unavailable. Summary of the invention

[0004] In view of this, the present invention provides a distributed database supporting a massive version of metadata storage method and retrieval method to solve the problems of load imbalance, performance degradation, and service unavailability faced by large-capacity tables in distributed key-value databases when storing massive versions of metadata records.

[0005] In a first aspect, the present invention provides a method for storing metadata of a large number of versions supported by a distributed database, the method comprising:

[0006] Obtain initial metadata and a timestamp corresponding to the initial metadata, where the initial metadata includes an initial key-value pair consisting of an initial key and an initial value;

[0007] Add the timestamp as version information to the initial key, modify the initial key to obtain the modified target key, and combine the target key and the value corresponding to the target key to form the modified metadata;

[0008] Based on the modified target key, the modified metadata is sorted by timestamp from large to small to construct a large-capacity table, which is used to represent a table storing metadata records in a distributed key-value database.

[0009] In the present invention, by adding the timestamp as version information to the metadata key, and sorting the timestamps from large to small to form a new large-capacity table, it is easy to retrieve metadata records. It has the advantage of wide applicability and is applicable to most distributed key-value databases. It only needs the database to have the ability to sort in the lexicographic order of the keys. Without affecting the functions and performance of the distributed key-value database itself, it is achieved that there is no upper limit on the number of versions of metadata records supported. The retrieval performance and service quality are not affected by the number of versions of metadata records, which solves the problems of load imbalance, performance degradation, service unavailability, etc. faced by large-capacity tables in distributed key-value databases when storing massive versions of metadata records.

[0010] In an optional implementation, a timestamp is added to an initial key as version information, and the initial key is modified to obtain a modified target key, including:

[0011] Combine the initial key with the preset characteristic separator, and the difference between the maximum timestamp corresponding to the combined timestamp and the current timestamp to obtain the modified current target key.

[0012] In this method, a new key is constructed by adding a timestamp representing the version information to the key and splitting it with a special separator. This ensures that the latest version of the key is sorted at the front of other versions of the key in a large-capacity table. The database only needs to be able to sort in lexicographical order of the keys, which greatly facilitates subsequent retrieval.

[0013] In a second aspect, the present invention provides a distributed database supporting a massive version metadata retrieval method, the method comprising:

[0014] Obtaining a large-capacity table, where the large-capacity table is constructed by the distributed database supporting massive version metadata storage method according to any one of the first aspects;

[0015] Get the key search range, the key search range includes the start key to the end key;

[0016] According to the preset conditions, the large-capacity table is scanned starting from the start key, and the target version key that meets the preset conditions corresponding to each key in the key search range is screened until the end key is scanned, and the set of target version keys corresponding to each key in the key search range is obtained. The target version key and the target version value corresponding to the target version key are combined to obtain a target version key-value pair, and the set of target version key-value pairs is determined as the query result.

[0017] In the present invention, by taking the target version key and the corresponding key-value pair in each key as the query result, without traversing the keys of other versions, the retrieval performance and service quality are not affected by the number of versions of metadata records, and the problems of retrieval timeout and database server blockage when there are massive versions of metadata records are well solved. It not only meets the storage requirements of massive versions of metadata records, but also meets the retrieval requirements of massive versions.

[0018] In an optional implementation, screening to obtain a target version key that meets a preset condition corresponding to each key in the key search range until the end key is scanned includes:

[0019] Determine that the first version of the key corresponding to the current key is the latest version key;

[0020] Determine whether the latest version key meets the preset conditions;

[0021] When the latest version key meets the preset conditions, the latest version key is determined to be the target version key, all remaining version keys of the current key are skipped, and the first version key of the next key is scanned until the next key is the termination key, and the scanning ends.

[0022] In this method, the scan starts from the specified starting key, and the current key-value pair is filtered to see if it meets the conditions. If so, it is placed in the query result, and the remaining versions of the current key are skipped. It is determined whether the end key is scanned. If so, the scan is terminated and the query results are returned. No matter how many versions there are, you only need to scan and filter the first version according to the conditions, and other versions can be skipped directly. While the performance is greatly improved, the impact on the database server is minimized.

[0023] In an optional embodiment, the method further includes:

[0024] In response to the first query instruction, query the value corresponding to the first target key of the preset version in the large-capacity table to obtain the first target value;

[0025] In response to the second query instruction, query the second target value corresponding to the second target key of the latest version in the large-capacity table;

[0026] In response to the third query instruction, the bulk table is queried for a set of third key-value pairs of all versions of all third target keys that meet the second preset condition within the second preset key retrieval range.

[0027] In this way, all kinds of search needs are fully met. Using the constructed large-capacity table for search can greatly improve the performance of search needs and minimize the impact of search on the database server.

[0028] In a third aspect, the present invention provides a distributed database supporting a massive version metadata storage device, the device comprising:

[0029] A data acquisition module, used to acquire initial metadata and a timestamp corresponding to the initial metadata, wherein the initial metadata includes an initial key-value pair consisting of an initial key and an initial value;

[0030] The metadata modification module is used to add the timestamp as version information to the initial key, modify the initial key to obtain the modified target key, and combine the target key and the value corresponding to the target key to form the modified metadata;

[0031] The large-capacity table construction module is used to sort the modified metadata from large to small by timestamp based on the modified target key to construct a large-capacity table. The large-capacity table is used to represent a table storing metadata records in a distributed key-value database.

[0032] In a fourth aspect, the present invention provides a distributed database supporting a massive version metadata retrieval device, the device comprising:

[0033] A large capacity table acquisition module, used to acquire a large capacity table, the large capacity table is constructed by the distributed database metadata storage device of the third aspect;

[0034] A search range acquisition module is used to acquire a key search range, where the key search range includes a start key to an end key;

[0035] The data retrieval module is used to scan the large-capacity table starting from the starting key according to the preset conditions, screen out the target version key that meets the preset conditions corresponding to each key in the key retrieval range, until the end key is scanned, and obtain the set of target version keys corresponding to each key in the key retrieval range, combine the target version key and the target version value corresponding to the target version key to obtain a target version key-value pair, and determine the set of target version key-value pairs as the query result.

[0036] In a fifth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the distributed database supporting massive version metadata storage method of the first aspect or any corresponding embodiment thereof, or to execute the distributed database supporting massive version metadata retrieval method of the second aspect or any corresponding embodiment thereof.

[0037] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the distributed database supporting massive versions of metadata storage method of the first aspect or any corresponding embodiment thereof, or to execute the distributed database supporting massive versions of metadata retrieval method of the second aspect or any corresponding embodiment thereof.

[0038] In a seventh aspect, the present invention provides a computer program product, comprising computer instructions, the computer instructions being used to enable a computer to execute the distributed database supporting massive versions of metadata storage method of the first aspect or any corresponding embodiment thereof, or to execute the distributed database supporting massive versions of metadata retrieval method of the second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flowchart of a method for storing metadata of a large number of versions supported by a distributed database according to an embodiment of the present invention.

[0041] Figure 2 The diagram is a schematic diagram of the table structure and distribution of each metadata record in an HBase table according to an embodiment of the present invention.

[0042] Figure 3 It is a schematic diagram of the structure of each metadata record table in an HBase table according to an embodiment of the present invention.

[0043] Figure 4 The present invention is a flowchart of a distributed database supporting a method for retrieving metadata of a large number of versions according to an embodiment of the present invention.

[0044] Figure 5The figure is a schematic diagram of the principle of fast screening by a filter according to an embodiment of the present invention.

[0045] Figure 6 It is a structural block diagram of a distributed database supporting a massive version metadata storage device according to an embodiment of the present invention.

[0046] Figure 7 It is a structural block diagram of a distributed database supporting massive version metadata retrieval device according to an embodiment of the present invention.

[0047] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] In the related technology, a distributed key-value database is generally used to store this large capacity table. This is a non-relational database, and each metadata record is a key-value pair.<key,value> , the key is unique, and the dictionary is sorted with the key as the index. As the table capacity grows, the entire table will be divided into several partitions according to the key. Each partition contains a continuous key-value pair. When the capacity of a partition exceeds a certain threshold, the partition will be split into two sub-partitions, which are evenly distributed on different servers of the database. Distributed key-value databases better meet the various requirements of massive data storage systems for metadata storage. However, when some keys have massive versions, all versions of a key will be concentrated in a partition and cannot be divided. This will cause a partition to be extremely large, which will not only destroy the load balancing of the database, but in extreme cases, it will also cause the database servers where these partitions are located to be unavailable.

[0050] In order to solve the above problems, a distributed database supporting massive version metadata storage method and retrieval method are provided in the embodiment of the present invention, which are used in computer devices. It should be noted that the execution subject can be a distributed database supporting massive version metadata storage device and a distributed database supporting massive version metadata retrieval device, which can be implemented as part or all of the computer device through software, hardware or a combination of software and hardware. The computer device can be a terminal or a client or a server, and the server can be a server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer and other intelligent hardware devices. In the following method embodiments, the execution subject is taken as an example of a computer device for explanation.

[0051] The computer device in this embodiment is suitable for use scenarios of massive data storage in the fields of cloud computing-cloud base-storage and cloud computing-cloud base-database, where the metadata records stored have massive versions. The present invention provides a method for storing massive versions of metadata in a distributed database, by adding a timestamp as version information to the key of the metadata, and sorting the timestamps from large to small to form a new large-capacity table, so as to facilitate the retrieval of metadata records. It has the advantage of wide applicability and is suitable for most distributed key-value databases, as long as the database has the ability to sort according to the lexicographic order of the keys. Without affecting the functions and performance of the distributed key-value database itself, it is achieved that there is no upper limit on the number of versions of metadata records supported. The retrieval performance and service quality are not affected by the number of versions of the metadata records, which solves the problems of load imbalance, performance degradation, service unavailability, etc. faced by large-capacity tables in distributed key-value databases when storing massive versions of metadata records.

[0052] According to an embodiment of the present invention, an embodiment of a distributed database supporting a method for storing massive versions of metadata is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0053] In this embodiment, a distributed database supports a method for storing metadata of a large number of versions, which can be used in the above-mentioned computer device. Figure 1 is a flow chart of a method for storing metadata of a large number of versions supported by a distributed database according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0054] Step S101: Acquire initial metadata and a timestamp corresponding to the initial metadata.

[0055] In the embodiment of the present invention, the initial metadata includes an initial key-value pair consisting of an initial key and an initial value.

[0056] Step S102, adding the timestamp as version information to the initial key, modifying the initial key to obtain a modified target key, combining the target key and the value corresponding to the target key to form the modified metadata.

[0057] Step S103 , based on the modified target key, the modified metadata is sorted from large to small by timestamp to construct a large-capacity table.

[0058] In the embodiment of the present invention, the large-capacity table is used to represent a table storing metadata records in a distributed key-value database.

[0059] In an optional implementation, screening to obtain a target version key that meets a preset condition corresponding to each key in the key search range until the end key is scanned includes:

[0060] Step a1, determining that the first version of the key corresponding to the current key is the latest version key.

[0061] Step a2, determining whether the latest version key meets the preset conditions.

[0062] Step a3, when the latest version key meets the preset conditions, determine the latest version key as the target version key, skip all remaining version keys of the current key, scan the first version key of the next key, and end the scan when the next key is the termination key.

[0063] In one example, in a distributed key-value database, a metadata record is represented as a key-value pair<key,value> , each key-value pair has a timestamp ts, ts represents the version information of this key-value pair. When a key has a large number of versions, each version has different ts and value. In order to enable different versions of the same key to be divided into different partitions, the metadata record construction method proposed in this scheme is to add ts representing the version information to the key to form a new key'; and in order to meet the retrieval requirements, different versions of the same key should be sorted from large to small according to ts in the table, that is, the latest version is at the front of other versions of the key in the table, and the table itself is sorted from small to large according to the lexicographical order of the key, so the new key' is composed of: {key+feature separator S+(maximum value of ts-ts)}. Each version of the same key can be represented as a new key-value pair<key’,value> , each key' is different.

[0064] In one implementation scenario, taking HBase table as an example, Figure 2Schematic diagram of the structure and distribution of each metadata record in an HBase table according to an embodiment of the present invention. Figure 2 As shown, each metadata record in the HBase table is a key-value pair<key,value> A table can be divided into several partitions (Regions) by key, and each partition contains a continuous key-value pair. This partition is evenly distributed on the distributed key-value database server (RegionServer) cluster.

[0065] The following example uses five keys, each with 1 million versions, to illustrate the construction of metadata records. Assume that the five keys are:

[0066] {aaa, bbb, ccc, ddd, eee, fff}

[0067] The version of each key is a timestamp starting from 1000000. For each key, the timestamps ts of 1 million versions are:

[0068] {1000000, 1000001, 1000002, 1000003, …, 1999999}

[0069] Assuming that the maximum timestamp is 9999999 and the special separator is '_', then according to the construction method of metadata record, the new key' corresponding to each version of key = key + '_' + (9999999-ts). Taking key = aaa as an example, the new keys' corresponding to its 1 million versions are sorted in lexicographic order from small to large:

[0070] {aaa_8000000, aaa_8000001, aaa_8000002,…, aaa_8999999}

[0071] Figure 3 is a schematic diagram of the structure of each metadata record table in an HBase table according to an embodiment of the present invention, such as Figure 3 As shown in the figure, all versions of these five keys are stored as 5 million key-value pairs in the HBase table. These key-value pairs are sorted in lexicographic order from small to large. The key-value pairs corresponding to different versions of the same key are located continuously in the table. The entire table is divided into multiple regions, and each region is a continuous key-value pair. Through this metadata record construction method, no matter how many versions a key has, the entire table can be evenly divided into multiple regions, so that the load of the entire table can be balanced to the entire distributed key-value database cluster, and the number of key versions is no longer limited.

[0072] The distributed database provided in this embodiment supports a method for storing metadata with a large number of versions. By adding the timestamp as version information to the metadata key, and sorting the timestamp from large to small to form a new large-capacity table, it is easy to retrieve the metadata record. It has the advantage of wide applicability and is applicable to most distributed key-value databases. It only needs the database to have the ability to sort according to the lexicographic order of the key. Without affecting the function and performance of the distributed key-value database itself, it realizes that there is no upper limit on the number of versions of the supported metadata records. The retrieval performance and service quality are not affected by the number of versions of the metadata record, which solves the problems of load imbalance, performance degradation, service unavailability, etc. faced by the large-capacity table in the distributed key-value database when storing a large number of versions of metadata records. By starting the scan from the specified starting key, filtering and matching whether the current key-value pair meets the conditions, if so, it is placed in the query result, and skipping the other remaining versions of the current key, judging whether the end key is scanned, if so, the scan is terminated, and the query result is returned. No matter how many versions there are, it is only necessary to scan and filter the first version according to the conditions, and other versions can be directly skipped. While the performance is greatly improved, the impact on the database server is also minimized.

[0073] In this embodiment, a distributed database supports a method for retrieving metadata of a large number of versions, which can be used in the above-mentioned computer device. Figure 4 FIG. 1 is a flowchart of another distributed database supporting a massive version metadata retrieval method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0074] Step S401, obtaining a large capacity table.

[0075] In the embodiment of the present invention, the large-capacity table is constructed by the above-mentioned distributed database supporting the massive version metadata storage method.

[0076] Step S402, obtaining the key search range.

[0077] In the embodiment of the present invention, the key search range includes the start key to the end key.

[0078] Step S403, according to the preset conditions, scan the large-capacity table starting from the start key, filter out the target version key that meets the preset conditions corresponding to each key in the key search range, until scanning to the end key, obtain the set of target version keys corresponding to each key in the key search range, combine the target version key and the target version value corresponding to the target version key to obtain a target version key-value pair, and determine the set of target version key-value pairs as the query result.

[0079] In an optional implementation, screening to obtain a target version key that meets a preset condition corresponding to each key in the key search range until the end key is scanned includes:

[0080] Step b1, determining that the first version of the key corresponding to the current key is the latest version key.

[0081] Step b2, determining whether the latest version key meets the preset conditions.

[0082] Step b3, when the latest version key meets the preset conditions, determine the latest version key as the target version key, skip all remaining version keys of the current key, scan the first version key of the next key, and end the scan when the next key is the termination key.

[0083] In this method, the scan starts from the specified starting key, and the current key-value pair is filtered to see if it meets the conditions. If so, it is placed in the query result, and the remaining versions of the current key are skipped. It is determined whether the end key is scanned. If so, the scan is terminated and the query results are returned. No matter how many versions there are, you only need to scan and filter the first version according to the conditions, and other versions can be skipped directly. While the performance is greatly improved, the impact on the database server is minimized.

[0084] In an optional embodiment, the method further includes:

[0085] Step c1, in response to a first query instruction, query the value corresponding to the first target key of a preset version in the large-capacity table to obtain a first target value.

[0086] Step c2, in response to the second query instruction, query the second target value corresponding to the second target key of the latest version in the large-capacity table.

[0087] Step c3, in response to the third query instruction, query the large capacity table for a set of third key-value pairs of all versions of all third target keys that meet the second preset condition within the second preset key retrieval range.

[0088] In this way, all kinds of search needs are fully met. Using the constructed large-capacity table for search can greatly improve the performance of search needs and minimize the impact of search on the database server.

[0089] In one example, the retrieval requirements for metadata records are mainly divided into two categories: select and list. Select means querying the value corresponding to a key, which can be divided into two categories: querying the value corresponding to a specified version ts, and querying the value corresponding to the latest version without specifying version information. List is a batch query interface that queries the set of key-value pairs that meet the conditions within a specified key range [startKey, endKey]. In this solution, according to the multi-version situation, the list is divided into two categories: querying the set of key-value pairs of all versions of all keys that meet the conditions within a specified key range, and querying the set of the latest versions of all keys that meet the conditions within a specified key range.

[0090] After the metadata record construction method adds version information to the key, there are three types of retrieval requirements that can be met through the inherent characteristics of the distributed key-value database. These three types of retrieval requirements are two types of retrieval requirements corresponding to select and one retrieval requirement in list: querying the key-value pair set of all versions of all keys that meet the conditions within a specified key range. The last type of retrieval requirement cannot directly retrieve the results, and may even cause retrieval timeouts and database server congestion. To this end, a metadata record retrieval method is proposed, which can greatly improve the performance of the last type of retrieval requirement and minimize the impact of retrieval on the database server.

[0091] The retrieval method for metadata records is to design a fast filtering rule on the database server side, and perform efficient filtering within the specified key range [startKey, endKey] to select the latest versions of all keys that meet the conditions. First, scan the table sequentially starting from startKey. Since the latest version of each key is always ranked before other versions of the key, when the first version of a key is scanned, the version is the latest version of the key. Check whether the version meets the conditions. If it meets the conditions, put it in the result set, then skip all remaining versions of the key, jump directly to the first version of the next key and start scanning again. When the scan reaches endKey, the scan ends, and the key-value pairs in the result set are the query results. The advantages of this retrieval method are obvious. For each key, no matter how many versions it has, you only need to scan and filter the first version according to the conditions, and other versions can be skipped directly. While the performance is greatly improved, the impact on the database server is also minimized.

[0092] In an implementation scenario, when retrieval is required, there are four types of retrieval requirements:

[0093] 1. Query the value corresponding to a specified version ts of a key.

[0094] 2. Query the value corresponding to the latest version of a key.

[0095] 3. Query the key-value pair set of all versions of all keys that meet the conditions within a specified key range [startKey, endKey].

[0096] 4. Query the latest version of all keys that meet the conditions within a specified key range.

[0097] In HBase, the first three search requirements can be easily met through its own characteristics, so this solution will not be described in detail. For the fourth search requirement, a filter is designed to quickly filter the key-value pairs that meet the conditions on the database server. Figure 5 FIG. 1 is a schematic diagram of the principle of fast screening by a filter according to an embodiment of the present invention. Figure 5 As shown, the Filter starts scanning from the specified starting key and filters the current key-value pairs.<key,value> Check whether the condition is met. If so, put it in the result list and skip the remaining versions. Determine whether the scan reaches the endKey. If so, end the scan and return the result list. Otherwise, continue to repeat the above scanning process.

[0098] The distributed database provided in this embodiment supports a method for retrieving metadata with a large number of versions. By taking the key of the target version in each key and the corresponding key-value pair as the query result, the keys of other versions are not traversed, so that the retrieval performance and service quality are not affected by the number of versions of metadata records, and the problems of retrieval timeout and blocking of the database server when there are a large number of versions of metadata records are well solved. It not only meets the storage requirements of a large number of versions of metadata records, but also meets the retrieval requirements of a large number of versions.

[0099] In this embodiment, a distributed database supporting a massive version of metadata storage device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0100] This embodiment provides a distributed database supporting a massive version of metadata storage device, such as Figure 6 As shown, including:

[0101] The data acquisition module 601 is used to acquire the initial metadata and the timestamp corresponding to the initial metadata. The initial metadata includes an initial key-value pair consisting of an initial key and an initial value. Figure 1Step S101 of the illustrated embodiment will not be described in detail here.

[0102] The metadata modification module 602 is used to add the timestamp as version information to the initial key, modify the initial key, obtain the modified target key, and combine the target key and the value corresponding to the target key to form the modified metadata. For details, please refer to Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0103] The large capacity table construction module 603 is used to sort the modified metadata from large to small by timestamp based on the modified target key to construct a large capacity table. The large capacity table is used to represent the table storing metadata records in the distributed key-value database. For details, see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0104] In some optional implementations, the metadata modification module 602 includes:

[0105] The metadata modification unit is used to combine the initial key with the preset characteristic separator, the difference between the maximum timestamp corresponding to the combined timestamp and the current timestamp, to obtain the modified current target key.

[0106] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0107] The distributed database in this embodiment supports a massive version of metadata storage device in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0108] In this embodiment, a distributed database supporting massive version metadata retrieval device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0109] This embodiment provides a distributed database supporting a massive version metadata retrieval device, such as Figure 7 As shown, including:

[0110] The large capacity table acquisition module 701 is used to acquire a large capacity table, which is constructed by the distributed database supporting the massive version metadata storage device of the third aspect. Figure 4 Step S401 of the illustrated embodiment will not be described in detail here.

[0111] The search range acquisition module 702 is used to acquire the key search range, and the key search range includes the start key to the end key. Figure 4 Step S402 of the illustrated embodiment will not be described in detail here.

[0112] The data retrieval module 703 is used to scan the large capacity table starting from the start key according to the preset conditions, screen and obtain the target version key that meets the preset conditions corresponding to each key in the key retrieval range, until the end key is scanned, obtain the set of target version keys corresponding to each key in the key retrieval range, combine the target version key and the target version value corresponding to the target version key to obtain the target version key-value pair, and determine the set of target version key-value pairs as the query result. For details, please refer to Figure 4 Step S403 of the illustrated embodiment will not be described in detail here.

[0113] In some optional implementations, the data retrieval module 703 includes:

[0114] The latest version key determination unit is used to determine that the first version of the key corresponding to the current key is the latest version key.

[0115] The preset condition judgment unit is used to judge whether the latest version key meets the preset condition.

[0116] The remaining key skipping unit is used to determine the latest version key as the target version key when the latest version key meets the preset conditions, skip all remaining version keys of the current key, and scan the first version key of the next key until the next key is the termination key, ending the scan.

[0117] In some optional implementations, the distributed database supporting massive version metadata retrieval apparatus further includes:

[0118] The first retrieval unit is used to query the value corresponding to the first target key of the preset version in the large-capacity table in response to the first query instruction to obtain the first target value.

[0119] The second retrieval unit is used to query the second target value corresponding to the second target key of the latest version in the large-capacity table in response to the second query instruction.

[0120] The third retrieval unit is used to query the set of third key-value pairs of all versions of all third target keys that meet the second preset condition within the second preset key retrieval range in the large capacity table in response to the third query instruction.

[0121] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0122] The distributed database supporting massive version metadata retrieval device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0123] The embodiment of the present invention also provides a computer device having the above Figure 6 The distributed database shown supports a massive version metadata storage device and the above Figure 7 The distributed database shown supports a massive version metadata retrieval device.

[0124] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0125] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0126] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0127] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0129] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0130] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0131] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0132] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0133] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A distributed database supports a method for storing metadata of massive versions, characterized in that: The method comprises: Acquire initial metadata and a timestamp corresponding to the initial metadata, wherein the initial metadata includes an initial key-value pair consisting of an initial key and an initial value; Adding the timestamp as version information to the initial key, modifying the initial key to obtain a modified target key, and combining the target key and a value corresponding to the target key to form modified metadata; Based on the modified target key, the modified metadata is sorted from large to small according to the timestamp to construct a large-capacity table, where the large-capacity table is used to represent a table storing metadata records in a distributed key-value database.

2. The method according to claim 1, characterized in that The adding the timestamp as version information to the initial key, modifying the initial key, and obtaining a modified target key includes: The initial key is combined with a preset characteristic separator, and the difference between the maximum value of the timestamp corresponding to the timestamp and the current timestamp is combined to obtain a modified current target key.

3. A distributed database supports a massive version metadata retrieval method, characterized in that: The method comprises: Obtaining a large-capacity table, wherein the large-capacity table is constructed by the distributed database supporting massive version metadata storage method according to any one of claims 1 to 2; Obtaining a key search range, where the key search range includes a start key to an end key; According to the preset conditions, the large-capacity table is scanned starting from the start key, and the target version key that meets the preset conditions corresponding to each key in the key search range is screened until the end key is scanned, and a set of target version keys corresponding to each key in the key search range is obtained, and the target version keys and the target version values ​​corresponding to the target version keys are combined to obtain target version key-value pairs, and the set of target version key-value pairs is determined as the query result.

4. The method according to claim 3, characterized in that The screening obtains the target version key that meets the preset condition and corresponds to each key in the key search range until the termination key is scanned, including: Determine that the first version of the key corresponding to the current key is the latest version key; Determine whether the latest version key meets the preset condition; When the latest version key meets the preset condition, the latest version key is determined to be the target version key, all remaining version keys of the current key are skipped, and the first version key of the next key is scanned until the next key is the termination key, and the scanning is ended.

5. The method according to claim 3, characterized in that: The method further comprises: In response to the first query instruction, query the value corresponding to the first target key of the preset version in the large-capacity table to obtain the first target value; In response to the second query instruction, query the second target value corresponding to the second target key of the latest version in the large-capacity table; In response to the third query instruction, the large-capacity table is queried for a set of third key-value pairs of all versions of all third target keys that meet the second preset condition within the second preset key retrieval range.

6. A distributed database supports a massive version metadata storage device, characterized in that: The device comprises: A data acquisition module, used to acquire initial metadata and a timestamp corresponding to the initial metadata, wherein the initial metadata includes an initial key-value pair consisting of an initial key and an initial value; A metadata modification module, used to add the timestamp as version information to the initial key, modify the initial key to obtain a modified target key, and combine the target key and the value corresponding to the target key to form the modified metadata; A large capacity table construction module is used to sort the modified metadata from large to small according to the timestamp based on the modified target key to construct a large capacity table, wherein the large capacity table is used to represent a table storing metadata records in a distributed key-value database.

7. A distributed database supports a massive version metadata retrieval device, characterized in that: The device comprises: A large capacity table acquisition module, used to acquire a large capacity table, wherein the large capacity table is constructed by the distributed database supporting massive version metadata storage device according to claim 6; A search range acquisition module, used to acquire a key search range, wherein the key search range includes a start key to an end key; The data retrieval module is used to scan the large-capacity table starting from the start key according to preset conditions, screen and obtain the target version key that meets the preset conditions corresponding to each key in the key retrieval range until the end key is scanned, obtain the set of target version keys corresponding to each key in the key retrieval range, combine the target version key and the target version value corresponding to the target version key to obtain a target version key-value pair, and determine the set of target version key-value pairs as the query result.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the distributed database supporting massive version metadata storage method described in any one of claims 1 to 2 or the distributed database supporting massive version metadata retrieval method described in any one of claims 3 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the distributed database supporting massive versions of metadata storage method described in any one of claims 1 to 2 or the distributed database supporting massive versions of metadata retrieval method described in any one of claims 3 to 5.

10. A computer program product, characterized in that It includes computer instructions, which are used to enable a computer to execute the distributed database supporting massive versions of metadata storage method described in any one of claims 1 to 2 or the distributed database supporting massive versions of metadata retrieval method described in any one of claims 3 to 5.

Images