Mass metadata deep paging cache acceleration method based on interval density

Abstract

The invention discloses a massive metadata deep paging cache acceleration method based on interval density. The method is used for solving the problem of ordered retrieval efficiency under a massive data set. According to the method, global ordered deep paging of the mass metadata is obtained through three main stages of fine partition interval metadata ordered feature extraction, local aggregation paging and deep paging cache generation, and the method has the capability of quickly and orderly retrieving the mass metadata with the magnitude of 10 billions, and can be used for quickly and orderly retrieving the mass metadata with the magnitude of 10 billions under the condition of database nodes with limited memories. And deep paging and ordered retrieval are performed on the mass metadata far exceeding the memory capacity, so that the method has the advantage that the ordered retrieval time of the mass data is constant.

Description

Technical Field

[0001] The present invention belongs to the field of deep paging optimization algorithms in a massively parallel processing database MMP, and specifically is a mass metadata deep paging cache acceleration method based on interval density. Background Art

[0002] With the development of the times and the advancement of technology, the types of various sensor equipment are becoming more and more abundant, and the accuracy of the equipment is getting higher and higher. Especially in the field of ship testing, the number of high-precision sensors equipped with various test equipment is increasing. These sensors not only have significantly improved measurement accuracy, but also greatly increased measurement frequency, which leads to a sharp increase in the amount of measurement data.

[0003] In order to effectively manage and use the massive data generated by various high-precision sensors, the massively parallel processing (MMP) database has gradually become the mainstream technical architecture. In the MMP database, metadata management follows the basic principles and design algorithms of the database to index and distribute the metadata. However, for users, metadata retrieval often requires orderly retrieval of a certain attribute group of specific metadata according to specific application logic, which involves global sorting of metadata under limited computing resources. When the amount of data is too large, the hardware CPU and memory will become bottlenecks. When sorting and retrieving global data, the retrieval efficiency will drop rapidly as the amount of data increases, and even cause node resource alarm failures. For this reason, deep paging technology needs to be used for processing. This is a challenging task for massive data sets. It is necessary to ensure that the orderly retrieval of data is completed, and the time cost is fast enough and will not increase linearly with the growth of data, so as to make the retrieval time as constant as possible. Summary of the invention

[0004] To solve the above problems, the inventor provides a method for accelerating the deep paging cache of massive metadata based on interval density to solve the problem of orderly retrieval efficiency under massive data sets. The method obtains global ordered deep paging of massive metadata through three main stages: precise interval metadata ordered feature extraction, local aggregation paging, and deep paging cache generation. The method has the ability to quickly and orderly retrieve massive metadata of the order of tens of billions. It can perform deep paging and orderly retrieval of massive metadata far exceeding the memory capacity under the condition of memory-constrained database nodes, and has the advantage of constant ordered retrieval time for massive data.

[0005] The present invention is achieved in that:

[0006] A method for accelerating deep paging cache of massive metadata based on interval density comprises the following steps:

[0007] Step S1, abstracting metadata interval categories from application business logic, and recording them in sequence, and finally forming a metadata interval set meta_interval_set;

[0008] Step S2, traverse each element in meta_interval_set in turn, and for each category, mark the ordered feature group definition of the category to form an ordered metadata feature set ordered_meta_set;

[0009] Step S3: query the database to count the combinations of elements in the sets in steps S1 and S2.

[0010] <meta_interval_set(i),ordered_meta_set(i)> The number of metadata items and sorting;

[0011] Step S4: According to the result of step S3, the metadata density of each interval category is calculated, and then processed according to the number N of single optimal data sorting performance of the actually deployed MMP database. The rule is: if the metadata density is low, the intervals are merged in sequence; if the metadata density is high, the intervals are split in sequence, and finally a metadata interval set uniform_meta_set with uniform density is formed;

[0012] Step S5, create a new cache table in the MMP database, and then generate a deep paging cache deep_pagination_cache based on uniform_meta_set in step S4 and some attributes of the metadata;

[0013] Step S6: When the application layer business logic performs an orderly search for metadata, it first accesses the deep_pagination_cache in step S5, and then queries the metadata from the MMP database.

[0014] Furthermore, the metadata interval category in step S1 refers to a division in the metadata that classifies and aggregates a specific attribute (or attribute group) of the metadata, and a set consisting of multiple different divisions is the metadata interval set meta_interval_set;

[0015] For example, suppose the metadata MetaData has N attributes, and each time n attributes (n∈N) are taken as the basis for dividing the metadata interval category, then each element in the metadata interval set meta_interval_set i It is a specific group of attributes, such as:

[0016] meta_interval_set1={{Priority1, Priority2,...Priority n1}}

[0017] meta_interval_set2={{Priority1, Priority2,...Priority n2}} ...

[0019] meta_interval_set i ={{Priority1, Priority2,...Priority ni}}

[0020] n1,n2..ni∈N

[0021] The collection of these attribute groups constitutes meta_interval_set, namely:

[0022] meta_interval_set=meta_interval_set1∪meta_interval_set2∪...meta_interval_set i Let's take a special example to illustrate the above set. For example, if the metadata is transportation data and the attributes are (driving speed, driving mode, fuel type), then (driving mode) can be selected as a specific attribute group, such as (driving mode = flight) to distinguish the data of all aircraft.

[0023] Furthermore, the ordered feature group definition in step S2 refers to the data column tuple that needs to be used for sorting in the metadata interval category. When the metadata is retrieved in an ordered manner, the data column tuple will be sorted according to the order of the columns in the data column tuple.

[0024] The ordered feature group definition clarifies the priority of data sorting. When the definition has multiple columns, they will be sorted in the order of the columns. For example, the metadata takes vehicle data as an example. The ordered feature group definition is (driving speed in increasing order, vehicle number in decreasing order). Then, they will be sorted in the order of increasing driving speed. If the driving speeds are the same, they will be sorted in the order of decreasing vehicle number. At this time, the "driving speed in increasing order" in front is the first priority, and the "vehicle number in decreasing order" is the second priority.

[0025] Furthermore, in step S3, the element combination<meta_interval_set(i),ordered_meta_set(i)> The meaning is: after the metadata is classified and divided according to a certain attribute group, it is sorted according to the feature group and the total number of data items is counted;

[0026] For example, using the vehicle data, meta_interval_set(i) is the attribute group {{driving mode = flight, fuel type = gasoline}}, and the feature group is: (production date ascending), then this step will count the number of data items for each production date in ascending order of production date for vehicles that fly and use gasoline;

[0027] The element combination is formed according to the application logic of the business, and it is not necessary to perform data statistics on all combinations of metadata interval categories and ordered feature groups;

[0028] For example, if the application logic focuses on aircraft fault data, then attribute groups related to fault diagnosis, such as (number of maintenance times, years of use), will be used and sorted and counted in ascending order of years of use, rather than counting other combinations.

[0029] Furthermore, the metadata density in step S4 refers to the percentage of data of the interval category in each fixed unit number of data, when the metadata is sorted according to the default index in the MMP database under the current specific metadata interval category;

[0030] Because the database will manage metadata when it is stored in the database, the database will usually generate a series of auxiliary storage and retrieval structures based on the table definition, such as primary keys or indexes. These structures are not necessarily consistent with the application logic during ordered retrieval. Therefore, in order to load application-related data at one time when loading data, it is necessary to first calculate the distribution density of metadata under specific application logic. For example, according to the order of database storage, among every one million data, there are 100,000 data with attribute A, accounting for 10%. Only by calculating this density can we provide a basis for subsequent interval merging or splitting.

[0031] The number of data items with optimal sorting performance at a single time, N, is an empirical value that depends on the software structure and hardware performance of the MMP database when it is deployed. It represents the number of data items that are processed at a moderate performance and resource usage when sorting metadata on a database node.

[0032] Since the performance evaluation of the database system is an overall evaluation, the performance cannot be defined by simply looking at the speed of a single query. It is necessary to comprehensively consider the CPU usage ratio, memory usage ratio, and the number of concurrent users. Therefore, it is necessary to find a single maximum number of processed items N that achieves a balance between performance and resources, so as to achieve the best overall ordered retrieval performance of the database.

[0033] According to the calculated metadata density, the intervals formed by the positions of the metadata in the default index are merged or split in sequence. The principle of merging or splitting is: the metadata under the current specific metadata interval category is merged or split into orderly intervals in the database default index as much as possible. In each interval, the number of data items with the metadata interval category is as close to N as possible.

[0034] For example, there are 400,000 pieces of transportation metadata in the database, among which statistics are performed on the data with (driving mode = flying) sorted in ascending order by production date. Later, it is found that with a statistical density of every 200,000 pieces of data, the number of data that meet the conditions in the intervals [1-200000] and [200001-400000] are 50,000 and 60,000 respectively, while the optimal number of data items N is 200,000. Therefore, it is necessary to merge all the intervals to make them close to the optimal number of 200,000 for word processing. Conversely, the principle of splitting intervals is similar.

[0035] Furthermore, in step S5, the cache table is a separate database table, which is used to record the index upper and lower bounds of each metadata interval category in the metadata interval set uniform_meta_set. This record is used as key paging information during the database ordered search to implement the deep paging function of the ordered search;

[0036] The cache table formed in this step is independent of the metadata table. It records the upper and lower limit information of the index of each deep paging in the database. After the table is built once, it can be queried by the application multiple times. Since each data in the cache table represents an ordered index to N metadata (N is the optimal number of processing items in step S4), the retrieval time overhead of the cache table can be approximately regarded as constant.

[0037] Furthermore, in step S6, the application layer needs to make corresponding constraints on the ordered retrieval of metadata according to the algorithm of the present invention. First, the cache information of the data to be retrieved must be found in the deep_pagination_cache table, and after obtaining the corresponding upper and lower bound information, the metadata is queried from the metadata table.

[0038] By querying the cache table first and then the metadata table, you can directly filter out all other metadata that is not related to the business application logic, thereby greatly speeding up the ordered retrieval speed with more appropriate hardware resource overhead.

[0039] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the steps of a mass metadata deep paging cache acceleration method based on interval density described in the present invention.

[0040] The beneficial effects of the present invention are:

[0041] The present invention obtains global ordered deep paging of massive metadata through precise interval metadata ordered feature extraction, local aggregation paging, and deep paging cache generation steps. The method has the ability to quickly and orderly retrieve massive metadata of the order of tens of billions, and can perform ordered retrieval of massive metadata far exceeding the memory capacity under the condition of database nodes with limited memory, and has the advantage of constant ordered retrieval time for massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is an overall flow chart of the method of the present invention.

[0043] Figure 2 It is a schematic diagram of the principle of the method of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail with reference to the accompanying drawings and in combination with specific embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0045] Example 1

[0046] like Figure 1 and Figure 2 As shown, a massive metadata deep paging cache acceleration method based on interval density mainly includes three main stages: ordered feature extraction of metadata through precise intervalization, local aggregation paging, and deep paging cache generation.

[0047] The method comprises the following steps:

[0048] Step S1, abstracting metadata interval categories from application business logic, and recording them in sequence, and finally forming a metadata interval set meta_interval_set;

[0049] Step S2, traverse each element in meta_interval_set in turn, and for each category, mark the ordered feature group definition of the category to form an ordered metadata feature set ordered_meta_set;

[0050] Step S3: query the database to count the combinations of elements in the sets in steps S1 and S2.

[0051] <meta_interval_set(i),ordered_meta_set(i)> The number of metadata items and sorting;

[0052] Step S4: According to the result of step S3, the metadata density of each interval category is calculated, and then processed according to the number N of single optimal data sorting performance of the actually deployed MMP database. The rule is: if the metadata density is low, the intervals are merged in sequence; if the metadata density is high, the intervals are split in sequence, and finally a metadata interval set uniform_meta_set with uniform density is formed;

[0053] Step S5, create a new cache table in the MMP database, and then generate a deep paging cache deep_pagination_cache based on uniform_meta_set in step S4 and some attributes of the metadata;

[0054] Step S6: When the application layer business logic performs an orderly search for metadata, it first accesses the deep_pagination_cache in step S5, and then queries the metadata from the MMP database.

[0055] Example 2

[0056] If the MMP metadata set is vehicle data, the recorded content is the operation logs of various vehicles, and the business application logic is to sequentially retrieve the operation logs of land vehicles at a specific departure time, the metadata table table_meta_data structure and stored data are as follows:

[0057] table_meta_data

[0058] Serial Number ID Device Name Driving style Departure Time Arrival time Run log 1 Car 1 Land driving March 15 March 16 Log 1 2 Aircraft 1 Flying in the air March 2 March 2 Log 2 3 Ship 1 Driving in water March 3 March 6 Log 3 4 Aircraft 2 Flying in the air March 1 March 1 Log 4 5 Cars 2 Land driving March 12 March 13 Log 5 6 Cars 3 Land driving March 16 March 17 Log 6

[0059] The table structure is shown in the table above. To facilitate the explanation of the relevant process, the serial number ID in the table is assumed to be the index key used by the underlying database software to retrieve the data location. In actual applications, the implementation of the index will be more complicated.

[0060] The upper-level application business logic is: extract the operation logs of all land-based vehicles in ascending order of departure time.

[0061] According to step S1, metadata interval categories are abstracted from the application business logic and recorded in sequence to eventually form a metadata interval set meta_interval_set;

[0062] This time, the metadata has five attributes ("equipment name", "driving mode", "departure time", "arrival time", and "operation log") (the serial number ID is used as a database management structure and is not included in the metadata attribute). According to the application business logic, only one metadata interval category is taken, namely:

[0063] meta_interval_set1 = {{Driving mode = Land driving}}

[0064] At this time there is only one set, namely:

[0065] meta_interval_set = meta_interval_set1 = {{driving mode = land driving}}

[0066] In actual situations, the set may also include multiple attribute groups, such as {{travel mode = air flight, arrival time < March 20}, {travel mode = water travel}}. In the embodiment, it is simplified to one attribute for illustration.

[0067] According to step S2, traverse each element in meta_interval_set in turn, and for each category, mark the ordered feature group definition of the category to form an ordered metadata feature set ordered_meta_set;

[0068] By traversing meta_interval_set, it is found that there is only one element {driving mode = land driving}, and the ordered feature group of this category is defined as "(departure time increasing)", that is, ordered_meta_set = {(departure time increasing)}.

[0069] Here, the special case is simplified to one sorting attribute, so it only needs to be sorted in ascending order according to the departure time. In actual applications, there can be multiple attributes here, and these attributes have different priorities and should be sorted in order of priority.

[0070] According to step S3, query the database and count the combinations of elements in the sets in step S1 and step S2

[0071] <meta_interval_set(i),ordered_meta_set(i)> The number of metadata items and sorting;

[0072] At this time, according to the processing of the previous two steps, statistics are performed, that is, the number of data items of the combination of <{driving mode = land driving}, (departure time increasing)> is counted, and the data that meets the conditions are:

[0073]

[0074] Then sort it. After sorting, the metadata table becomes:

[0075] table_ordered_data

[0076] Serial Number ID Device Name Driving style Departure Time Arrival time Run log 1 Cars 2 Land driving March 12 March 13 Log 5 2 Aircraft 1 Flying in the air March 2 March 2 Log 2 3 Ship 1 Driving in water March 3 March 6 Log 3 4 Aircraft 2 Flying in the air March 1 March 1 Log 4 5 Car 1 Land driving March 15 March 16 Log 1 6 Cars 3 Land driving March 16 March 17 Log 6

[0077] Here, it is just a simple special case, which means that the application business logic only focuses on the data of <{driving mode = land driving}, (departure time increasing)>. In actual situations, due to the diversity of attribute groups and sorting groups, in actual situations, the element group here will have multiple rows.

[0078] According to step S4, the metadata density of each interval category is calculated based on the result of the previous step, and then processed according to the number N of single optimal data sorting performance of the actually deployed MMP database. The rule is: if the metadata density is low, the intervals are merged in sequence; if the metadata density is high, the intervals are split in sequence, and finally a metadata interval set uniform_meta_set with uniform density is formed;

[0079] Here, for the sake of convenience, assume that the number of database single-time better data sorting performance items N is 2. Therefore, table_ordered_data is used as the data table, and the process of calculating density is:

[0080] According to the "Serial Number ID" column of the database index management structure, the metadata is divided into three intervals, namely interval A: [1-2], B: [3-4], and C: [5-6]. Then, it is found that there is 1 data in interval A, 0 data in interval B, and 2 data in interval C that meet the business application logic. Therefore, according to the rules, the intervals are merged or split to make the amount of data processed at a time close to N = 2, then:

[0081] uniform_meta_set = {[1-5], [6-6]}, which involves the merging of intervals A, B, and C, and the splitting of interval C. In actual situations, due to the large amount of data, the number of data items after the merging and splitting of intervals may not be strictly equal to N. A value close to N can be selected according to actual conditions.

[0082] After this step, a source data interval set uniform_meta_set with uniform data density is obtained.

[0083] According to step S5, a new cache table is created in the MMP database, and then according to uniform_meta_set in step S4, combined with some attributes of the metadata, a deep paging cache deep_pagination_cache is generated;

[0084] At this time, create a new cache table deep_pagination_cache, and its structure is as follows:

[0085] Cache properties Retrieves the lower bound of a value attribute Retrieve the upper bound of a value attribute Index interval range

[0086] The cache attribute is the identifier of the page content, which is: <{Driving mode = Land driving}, (Departure time increasing)> in this example. For the convenience of subsequent explanation, it is abbreviated as (Land driving, Time increasing);

[0087] The upper limit of the search value attribute is the maximum value of the search field for paging, and the lower limit of the search value attribute is the minimum value of the search field for paging. In this example, the search field is "departure time";

[0088] The index interval range is: uniform_meta_set = {[1-5], [6-6]};

[0089] Therefore, in this step, the generated cache table deep_pagination_cache content is:

[0090] deep_pagination_cache

[0091]

[0092] According to the last step: step S6, when the application layer business logic performs an orderly search for metadata, the deep_pagination_cache in step S5 is first accessed, and then the metadata is queried from the MMP database.

[0093] At this time, assume that the business application logic is: collect the operation logs of land vehicles with departure time from March 12 to March 15, that is, collect them in order in the table table_ordered_data (log 5, log 1):

[0094] table_ordered_data

[0095] Serial Number ID Device Name Driving style Departure Time Arrival time Run log 1 Cars 2 Land driving March 12 March 13 Log 5 2 Aircraft 1 Flying in the air March 2 March 2 Log 2 3 Ship 1 Driving in water March 3 March 6 Log 3 4 Aircraft 2 Flying in the air March 1 March 1 Log 4 5 Car 1 Land driving March 15 March 16 Log 1 6 Cars 3 Land driving March 16 March 17 Log 6

[0096] The process is:

[0097] A. Search in deep_pagination_cache: cache attribute = (land line, time increment) record;

[0098] B. In the records retrieved by A, find the records where (the lower bound of the search value attribute is less than or equal to March 12) and (the upper bound of the search value attribute is greater than or equal to March 15). After searching, it is the first row of data:

[0099]

[0100] C. The index range read is: [1-5]

[0101] D. Then query from table_ordered_data, and the query condition directly specifies the index range: (serial number ID>=1) and (serial number ID<=5)

[0102] Qualitative analysis was performed according to this method:

[0103] Before using this method, querying metadata with a departure time in a specified range requires traversing the entire table and then sorting. However, after using the invented method, when querying metadata with a departure time in a specified range, each metadata interval that meets the conditions is directly located through the serial number ID index, and then the data in the interval is directly extracted.

[0104] The above example is only for explaining the principle. In actual applications, the number of metadata sets may reach tens of billions. According to hardware configuration and experience, the optimal number N of data to be processed at one time is generally 1 million. Therefore, it can be roughly assumed that every 1 million metadata will eventually form one cache data. In actual applications, the speed of ordered retrieval after using the present invention is orders of magnitude different from that of direct query.

[0105] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A method for accelerating deep paging cache of massive metadata based on interval density, characterized in that: The following steps are involved: S1. Metadata interval categories are abstracted from the application business logic and recorded in sequence to eventually form a metadata interval set meta_interval_set; S2. Traverse each element in meta_interval_set in turn, and for each category, mark the ordered feature group definition of the category to form an ordered metadata feature set ordered_meta_set; S3, query the database and count the combinations of elements in the sets in step S1 and step S2 <meta_interval_set(i),ordered_meta_set(i)> The number of metadata items and sorting; S4. According to the result of step S3, the metadata density of each interval category is calculated, and then processed according to the number N of single optimal data sorting performance of the actually deployed MMP database, and finally a metadata interval set uniform_meta_set with uniform density is formed; S5. Create a new cache table in the MMP database, and then generate a deep paging cache deep_pagination_cache based on uniform_meta_set in step S4 and some attributes of the metadata; S6. When the application layer business logic performs an orderly search for metadata, it first accesses the deep_pagination_cache in step S5, and then queries the metadata from the MMP database.

2. The method according to claim 1, characterized in that: The processing rule described in step S4 according to the number N of single-time optimal data sorting performance of the actually deployed MMP database is: if the metadata density is low, the intervals are merged in sequence; if the metadata density is high, the intervals are split in sequence.

3. The method according to claim 1, characterized in that: The metadata interval category in step S1 refers to a division in the metadata that classifies and aggregates a specific attribute or attribute group of the metadata. A set consisting of multiple different divisions is the metadata interval set meta_interval_set.

4. The method according to claim 1, characterized in that: The ordered feature group definition in step S2 refers to the data column tuple that needs to be used for sorting in the metadata interval category. When the metadata is retrieved in an ordered manner, the data column tuple will be sorted according to the order of the columns in the data column tuple.

5. The method according to claim 4, characterized in that: The order of the columns in the data column tuple is used for sorting. If there are multiple columns in the definition, the columns are sorted in sequence.

6. The method according to claim 1, characterized in that: The element combination in step S3<meta_interval_set(i),ordered_meta_set(i)> The meaning is: after the metadata is classified and divided according to a certain attribute group, it is sorted according to the feature group and the total number of data items is counted.

7. The method according to claim 1, characterized in that: The metadata density in step S4 includes: under the current specific metadata interval category, the metadata is sorted according to the default index in the MMP database, and the percentage of data of the interval category in each fixed unit number of data; According to the calculated metadata density, the intervals formed by the positions of the metadata in the default index are merged or split in sequence. The principle of merging or splitting is: the metadata under the current specific metadata interval category is merged or split into ordered intervals one by one in the database default index. In each interval, the number of data items with this metadata interval category approaches N.

8. The method according to claim 1, characterized in that: The cache table in step S5 is a separate database table, which is used to record the index upper and lower bounds of each metadata interval category in the metadata interval set uniform_meta_set. This record is used as key paging information during the ordered retrieval of the database to realize the deep paging function of the ordered retrieval; The cache table formed in this step is independent of the metadata table, and records the upper and lower limit information of the index of each deep page in the database. After the table is built once, it is available for multiple queries. Since each data in the cache table represents an ordered index to N metadata, the retrieval time overhead of the cache table is approximately considered constant.

9. The method according to any one of claims 1 to 8, characterized in that: In step S6, the application layer needs to make corresponding constraints on the ordered retrieval of metadata according to the algorithm of the present invention. First, the cache information of the data to be retrieved is searched in the deep_pagination_cache table, and after obtaining the corresponding upper and lower bound information, the metadata is queried from the metadata table. By first querying the cache table and then querying the metadata table, all other metadata that is not related to the business application logic can be directly filtered out, thereby greatly accelerating the orderly retrieval speed with more appropriate hardware resource overhead.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program can be executed by a processor to implement the steps of a massive metadata deep paging cache acceleration method based on interval density as described in any one of claims 1-9.

Images