Method for processing multi-dimensional ordered data using layered buckets and application

Abstract

A method, device, medium and electronic equipment for processing multi-dimensional ordered data by using hierarchical buckets are disclosed. The method comprises: parsing a hierarchical bucket configuration file to obtain bucketing strategy information of each bucket layer; constructing a bucket index mapper and a hierarchical bucket index tree according to the bucketing strategy information, and setting a storage container for each leaf bucket node in the hierarchical bucket index tree; storing the to-be-added multi-dimensional ordered data in the storage container of the leaf bucket node to which the multi-dimensional ordered data belongs according to the value of each sorting key in the to-be-added multi-dimensional ordered data, the bucket index mapper of each bucket layer and the hierarchical bucket index tree; and deleting the to-be-deleted multi-dimensional ordered data from the storage container of the leaf bucket node to which the multi-dimensional ordered data belongs according to the value of each sorting key in the to-be-deleted multi-dimensional ordered data and the bucket index mapper of each bucket layer in the hierarchical bucket index tree. The present disclosure can better adapt to application scenarios of high-frequency data update and high-frequency data access, and is conducive to reducing the maintenance cost of the system.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method for processing multidimensional ordered data using hierarchical buckets, an apparatus for processing multidimensional ordered data using hierarchical buckets, a storage medium, and an electronic device. Background Technology

[0002] Data processing typically includes operations such as adding, deleting, and traversing data. Currently, the following two methods are mainly used to process multidimensional ordered data:

[0003] Method 1: Using a doubly linked list to process multidimensional ordered data. Specifically, the doubly linked list is used as the storage container. When inserting data, starting from the head node, the data is compared dimension-by-dimensional with the data in each node according to the multidimensional sort key until the insertion position is found. The predecessor and successor pointers of the corresponding node are then adjusted to ensure the data is stored in the doubly linked list. Similarly, when deleting data, the process is repeated, starting from the head node, comparing the data dimension-by-dimensional with the data in each node according to the multidimensional sort key until the storage position is found. The predecessor and successor pointers of the corresponding node are then adjusted to ensure the doubly linked list no longer contains the data. Therefore, this method maintains the order of multidimensional ordered data through a full linear traversal.

[0004] Method 2: Utilize sequential containers to process multidimensional ordered data. Specifically, use arrays or dynamic data cache containers with contiguous storage space to store multidimensional ordered data. When inserting data, starting from the initial storage unit of the cache container, compare the data dimension by dimension with the data in the storage unit according to the multidimensional sort key until the insertion position of the data is found. Then, by shifting the data in the storage units after the insertion position, the data is stored at that insertion position in the storage container. Similarly, when deleting data, start from the initial storage unit of the cache container, compare the data dimension by dimension with the data in the node according to the multidimensional sort key until the storage position of the data is found. Then, by shifting the data in the storage units after the deletion position, the storage container no longer contains the data.

[0005] As described above, existing data processing methods maintain the sorting of multidimensional ordered data through a full linear traversal. This results in high location costs for operations such as insertion and deletion (e.g., the average time complexity of insertion and deletion operations is O(n)). Consequently, the cost of these operations is directly constrained by the data scale. In other words, as the data volume increases, full linear traversal leads to a significant decrease in system performance, making it unsuitable for applications requiring high-frequency data updates and access. Furthermore, the sorting rules and other components in existing data processing methods typically need to be set in the program code. Adjusting these rules requires modifying the program code, preventing dynamic and flexible adjustments based on actual needs. Therefore, how to avoid the problems of existing data processing methods and reduce the location costs of insertion and deletion operations while enabling dynamic and flexible adjustments to sorting rules based on actual needs is a technical issue worthy of attention. Summary of the Invention

[0006] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method for processing multidimensional ordered data using hierarchical bucketing, an apparatus for processing multidimensional ordered data using hierarchical bucketing, a storage medium, and an electronic device.

[0007] According to a first aspect of the present disclosure, a method for processing multidimensional ordered data using hierarchical bucketing is provided. The method includes: parsing hierarchical bucket configuration information in a hierarchical bucket configuration file and obtaining bucketing strategy information for each bucket layer based on the parsing result; wherein the number of hierarchical bucket layers is determined by the number of sorting keys in the multidimensional ordered data; constructing a bucket index mapper for each bucket layer according to the bucketing strategy information for each bucket layer; wherein the bucket index mapper constructed for any bucket layer corresponds to a sorting key in the multidimensional ordered data; setting the number of bucket nodes in each bucket layer of the hierarchical bucket index tree according to the bucketing strategy information for each bucket layer, and assigning bucket indexes to each bucket node in each bucket layer to generate a hierarchical bucket index tree. For each leaf bucket node in the hierarchical bucket index tree, a storage container is set up. For the multidimensional ordered data to be added, the leaf bucket node to which the multidimensional ordered data to be added belongs is determined according to the values ​​of each sort key in the multidimensional ordered data to be added, the bucket index mapper of each bucket layer, and the hierarchical bucket index tree, and the multidimensional ordered data to be added is stored in the storage container of its leaf bucket node. For the multidimensional ordered data to be deleted, the leaf bucket node to which the multidimensional ordered data to be deleted belongs is determined according to the values ​​of each sort key in the multidimensional ordered data to be deleted and the bucket index mapper of each bucket layer in the hierarchical bucket index tree, and the multidimensional ordered data to be deleted is deleted from the storage container of its leaf bucket node.

[0008] According to a second aspect of the present disclosure, an apparatus for processing multidimensional ordered data using hierarchical bucketing is provided. The apparatus includes: a parsing module, configured to parse hierarchical bucket configuration information in a hierarchical bucket configuration file and obtain bucketing strategy information for each bucket layer based on the parsing result; wherein the number of hierarchical bucket layers is determined by the number of sorting keys in the multidimensional ordered data; a mapper construction module, configured to construct a bucket index mapper for each bucket layer according to the bucketing strategy information for each bucket layer; wherein the bucket index mapper constructed for any bucket layer corresponds to a sorting key in the multidimensional ordered data; and an index tree construction module, configured to set the number of bucket nodes in each bucket layer of the hierarchical bucket index tree according to the bucketing strategy information for each bucket layer, and assign bucket indexes to each bucket node in each bucket layer to generate a hierarchical bucket index tree; and a container setting module. The module is used to set up storage containers for each leaf bucket node in the hierarchical bucket index tree constructed by the index tree construction module; the data insertion module is used to determine the leaf bucket node to which the new multidimensional ordered data belongs based on the values ​​of each sort key in the new multidimensional ordered data, the bucket index mappers of each bucket level, and the hierarchical bucket index tree, and to store the new multidimensional ordered data in the storage container of its leaf bucket node; the data deletion module is used to determine the leaf bucket node to which the deleted multidimensional ordered data belongs based on the values ​​of each sort key in the deleted multidimensional ordered data and the bucket index mappers of each bucket level in the hierarchical bucket index tree, and to delete the deleted multidimensional ordered data from the storage container of its leaf bucket node.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for implementing any of the methods described above.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement any of the methods described above.

[0011] The method, data processing apparatus, storage medium, and electronic device for processing multidimensional ordered data using hierarchical bucketing provided by embodiments of this disclosure, by setting a hierarchical bucket configuration file independent of the program file and setting hierarchical bucket configuration information in the bucket configuration file, can obtain the bucketing strategy information of each bucket layer through parsing operations, thereby separating the data logic from the program code. That is, updates to data-related business logic can be implemented by modifying the bucket configuration file without modifying the program file. By using the bucketing strategy information of each bucket layer to generate a bucket index mapper and a hierarchical bucket index tree, and by classifying each leaf bucket node in the hierarchical bucket index tree... By not setting separate storage containers for each element, all multidimensional ordered data can be stored in the corresponding leaf bucket nodes' storage containers in different regions. By setting bucket index mappers and hierarchical bucket index trees, the storage container to which the multidimensional ordered data belongs can be quickly located, thus limiting data search to a single leaf bucket. This avoids the phenomenon of full linear search and facilitates the transformation of multidimensional sorting location into layer-by-layer one-dimensional location. That is, whether adding new multidimensional ordered data, deleting or accessing existing multidimensional ordered data, no global data search is required, significantly reducing the data search scope and making the average time complexity of insertion and deletion operations O(1). Therefore, the technical solution provided in this disclosure is well-suited for application scenarios involving high-frequency data updates and access, ensuring system flexibility and scalability while maintaining high-frequency update and access efficiency, and reducing system maintenance costs.

[0012] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0013] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0014] Figure 1 This is a flowchart of an embodiment of the method for processing multidimensional ordered data using hierarchical buckets disclosed herein;

[0015] Figure 2 This is a schematic diagram of an embodiment of the hierarchical bucket index tree of this disclosure;

[0016] Figure 3 This is a schematic diagram illustrating one embodiment of the working process of the bucket index mapper of this disclosure;

[0017] Figure 4 This is a schematic diagram of an embodiment of the present disclosure for finding the insertion position in a doubly linked list;

[0018] Figure 5 This is a schematic diagram of an embodiment of inserting a data node in a doubly linked list according to the present disclosure;

[0019] Figure 6 This is a schematic diagram of an embodiment of the apparatus for processing multidimensional ordered data using hierarchical buckets disclosed herein;

[0020] Figure 7 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0021] Example embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure, and it should be understood that this disclosure is not limited to the example embodiments described herein.

[0022] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0023] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0024] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0025] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0026] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0027] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0029] The embodiments of this disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0030] Exemplary methods

[0031] Figure 1 This is a flowchart illustrating an embodiment of the method for processing multidimensional ordered data using hierarchical bucketing provided in this disclosure. Figure 1 The illustrated embodiment includes steps: S100, S101, S102, S103, S104, and S105. The following describes... Figure 1 Each step in the process will be explained in detail.

[0032] S100. Parse the hierarchical bucket configuration information in the hierarchical bucket configuration file, and obtain the bucketing strategy information of each bucket layer based on the parsing results.

[0033] The multidimensional ordered data in this disclosure refers to data composed of multiple dimensional fields, and multiple multidimensional ordered data can form a globally ordered dataset based on the sorting of multiple fields. The number of dimensions of the multidimensional ordered data in this disclosure is usually related to the application scenario of the program, and this disclosure does not limit the number of dimensions of the multidimensional ordered data.

[0034] The multidimensional ordered data in this disclosure typically includes multiple sort keys, and each sort key represents a specific sorting dimension of the multidimensional ordered data. The number of sort keys in the multidimensional ordered data is usually the same as the number of fields; that is, one field is one sort key. For example, (region, sales amount) formed by the two dimensions of region and sales amount constitutes two-dimensional ordered data, and this two-dimensional ordered data includes two sort keys: region and sales amount. However, this disclosure does not exclude the possibility that the number of sort keys in the multidimensional ordered data is less than the number of fields, meaning that some fields in the multidimensional ordered data are not sort keys.

[0035] In one example, the sorting key attribute in this disclosure can be a numerical attribute or a discrete label attribute, etc. A numerical attribute refers to an attribute that can be represented by a number (i.e., a number) and supports size comparison and ordered sorting. For example, age, price, time, score, length, and distance can all be considered sorting keys for numerical attributes. A discrete label attribute refers to a label attribute used for classification, and it does not inherently possess a natural order of magnitude. For example, city, product type, user status, and time period (such as morning, afternoon, and evening) can all be considered sorting keys for discrete label attributes. The specific type of sorting key for multidimensional ordered data in this disclosure is usually related to the application scenario of the program, and this disclosure does not limit the specific representation of the sorting key for multidimensional ordered data.

[0036] In one example, the data types (such as string, integer, and floating-point) of all fields in multidimensional ordered data can be the same or different. Similarly, the attributes of all sort keys in multidimensional ordered data can be the same or different; for example, all sort keys in multidimensional ordered data may be numeric, or all sort keys may be discrete label attributes; another example is that some sort keys in multidimensional ordered data are discrete label attributes, while some sort keys are numeric attributes. In a more typical example, multidimensional ordered data usually includes one or more discrete label attribute sort keys and one numeric attribute sort key, with the last sort key being a numeric attribute and the others being discrete label attribute sort keys.

[0037] The hierarchical bucket configuration file in this disclosure refers to a pre-set configuration file containing structured configuration information; that is, the hierarchical bucket configuration file is mainly used to record hierarchical bucket configuration information. The hierarchical bucket configuration information in this disclosure is mainly used to generate the bucket index mapper and the hierarchical bucket index tree. Furthermore, the hierarchical bucket configuration file of this disclosure is a file independent of the program code; therefore, this disclosure allows data logic to be decoupled from the program code. In one example, the hierarchical bucket configuration file of this disclosure can specifically be a file based on formats such as JSON (JavaScript Object Notation).

[0038] The hierarchical bucketing in this disclosure can be considered a multi-level nested, multi-branched bucketing system. That is, the hierarchical bucketing in this disclosure refers to a composite bucket with multiple bucket layers formed by nesting multiple buckets. The outermost bucket can be considered the largest bucket, and the innermost bucket can be considered the smallest bucket. There may be no intermediate bucket between the outermost and innermost buckets (i.e., the hierarchical bucketing in this disclosure includes two bucket layers), or there may be at least one intermediate bucket layer (i.e., the hierarchical bucketing in this disclosure includes at least three bucket layers). Furthermore, the number of bucket layers in the hierarchical bucketing in this disclosure is usually determined by the number of sorting keys (i.e., the dimensions of the data) in the multidimensional ordered data. This disclosure does not limit the number of bucket layers contained in the hierarchical bucketing.

[0039] In one example, the number of layers in all the hierarchical buckets can be the same as the number of sort keys in the multidimensional ordered data, and the number of outermost buckets is usually multiple, with the number of outermost buckets typically matching the number of values ​​for their corresponding sort keys. For example, for two-dimensional ordered data such as (region, sales amount), the hierarchical buckets of this disclosure can have two layers, with the outermost bucket corresponding to the sort key "region" (i.e., the corresponding region field), and the innermost bucket corresponding to the sort key "sales amount" (i.e., the first sales amount field), and the number of outermost buckets can be the same as the number of all different values ​​for the region sort key. Of course, this disclosure also does not exclude the case where the number of layers in all the hierarchical buckets is the number of sort keys in the multidimensional ordered data plus one, in which case the number of outermost buckets is one; for example, also for two-dimensional ordered data such as (region, sales amount), the hierarchical buckets of this disclosure can have three layers, with the outermost bucket corresponding to the multidimensional ordered data, and the number of outermost buckets always being one, the middle buckets corresponding to the sort key "region" (the number of middle buckets can be the same as the number of all different values ​​for the region sort key), and the innermost bucket corresponding to the sort key "sales amount".

[0040] In one example, the hierarchical bucket configuration information disclosed herein may include: global configuration information and bucket-level configuration information. The global configuration information applies to all bucket levels, while the bucket-level configuration information applies to specific bucket levels. This simplifies the hierarchical bucket configuration file while allowing for flexible customization of the configuration information for each bucket level.

[0041] In one example, the global configuration information of this disclosure may include: the sorting method (such as ascending or descending order) used for the sorting keys of all numerical attributes in multidimensional ordered data, and the quantization processing rules for fields of a predetermined type. The sorting method can be considered as the comparator configuration information. The predetermined data type can be a floating-point type, etc., and the quantization processing rules can be considered as the conversion rules for the values ​​of fields of the predetermined data type. For example, based on the precision requirements of floating-point data in the actual application scenario, there are conversion rules for converting floating-point data to integer data. In a more specific example, the conversion rules for converting floating-point data to integer data may include at least one of: multiplying by a predetermined factor, rounding based on a predetermined precision (such as rounding to two decimal places), rounding down, and rounding up. For example, when the number of decimal places in floating-point data is usually large, the conversion rule could be: first multiply the floating-point data by a predetermined factor, and then perform rounding to two decimal places, rounding down, or rounding up. This disclosure, by setting quantization processing rules, helps to ensure the accuracy of bucket index mapping.

[0042] In one example, the global configuration information in this disclosure can be partially recessed into the corresponding bucket layer configuration information. For instance, the sorting method used for each sorting key in multidimensional ordered data can be set in the bucket layer configuration information, thereby achieving the goal of different sorting methods corresponding to different sorting keys. Furthermore, the global configuration information in this disclosure may also include other information, such as the number of bucket layers. This disclosure does not limit the specific content included in the global configuration information.

[0043] In one example, the bucket layer configuration information disclosed herein may include: the total number of buckets in each bucket layer and the bucket index mapping information for each bucket layer. The bucket index mapping information is used to represent the correspondence between the corresponding sort key value and the bucket index, and the bucket index mapping information in the hierarchical bucket configuration file can be represented as a bucket index rule array (rules[]). The bucket index rule array may include multiple elements, and each element can represent a bucket index mapping method.

[0044] In one example, the bucket layer configuration information of this disclosure may further include: a first fallback strategy information indicating that at least one bucket layer's bucket index mapping has failed. This first fallback strategy information indicates the bucketing method for the sorting key of the multidimensional valid data when the bucket index mapper of one bucket layer cannot obtain the bucket index corresponding to the value of a sorting key of the multidimensional valid data. For example, for multidimensional ordered data without a corresponding bucket index in one bucket layer, this disclosure can directly use the bucket index of the fallback bucket in that bucket layer as its corresponding bucket index. The bucketing process performed based on the first fallback strategy information of this disclosure can be called fallback bucketing. By setting the first fallback strategy information, it is beneficial to avoid the phenomenon of failed bucket index allocation for multidimensional ordered data, thereby avoiding the risk of multidimensional ordered data being discarded and the program being interrupted, and thus improving the data integrity and robustness of the multidimensional ordered data processing system.

[0045] In one example, the bucket layer configuration information of this disclosure may further include: a second fallback strategy information where the bucket index mapping is successful but the search for the storage location of the multidimensional ordered data to be added fails. This second fallback strategy information indicates a processing method for allocating storage locations for the valid multidimensional data when the leaf bucket node to which the multidimensional ordered data to be added belongs is successfully determined, but the corresponding storage location in the storage container is not successfully obtained according to the preset insertion position calculation method. Specific examples can be found in the descriptions of subsequent embodiments. The storage location allocation processing based on the second fallback strategy information of this disclosure can be called fallback location processing. By setting the second fallback strategy information, it is beneficial to avoid the phenomenon of failure to allocate storage locations for multidimensional ordered data, thereby avoiding the risk of multidimensional ordered data being discarded and the program being interrupted, and thus improving the data integrity and robustness of the multidimensional ordered data processing system.

[0046] In one example, this disclosure can perform lexical analysis on the hierarchical bucket configuration information in the hierarchical bucket configuration file to obtain a lexical list (such as keywords, values, and value ranges), and perform syntactic analysis and semantic analysis based on the lexical list to obtain the bucketing strategy information for each bucket layer.

[0047] In a more specific example, this disclosure uses a predefined mapping specification as the basis for performing lexical analysis, syntax analysis, and semantic analysis. The mapping specification can be set according to actual needs. An example of the mapping specification can be shown in Table 1 below:

[0048] Table 1

[0049] unlimited Mapping of special values ​​(e.g., infinity) index index eq Equal to a specific value value, index index lt Less than the threshold value, index index lte Less than or equal to the threshold value, index index gt Greater than the threshold value, index index gte Greater than or equal to the threshold value, index index range Within the closed interval min, max, index index range_to_index Interval mapping by step size min, max, index_base, step index_base + (value-min) / step

[0050] In one example, a multidimensional ordered data set is defined, including a power field, which corresponds to a sort key for a numerical attribute. The correspondence between the values ​​of the sort key corresponding to the power field and the corresponding bucket indexes is shown in Table 2 below.

[0051] Table 2

[0052] UNLIMITED 0 >= 1,000,000,000 1 201..999,999,999 2 1..200 3-202 (Index = power + 2) 0 203 < 0 204

[0053] The hierarchical bucket configuration information in the JSON-formatted file set in Tables 1 and 2 above may include the following:

[0054] {

[0055] "bucket_count": 205,

[0056] "unlimited_value": 9223372036854775807,

[0057] "compare": { "order": "asc"},

[0058] "rules": [

[0059] { "type": "unlimited", "index": 0},

[0060] { "type": "gte", "value": 1000000000, "index": 1},

[0061] { "type": "range", "min": 201, "max": 999999999, "index": 2},

[0062] { "type": "range_to_index", "min": 1, "max": 200, "index_base":3, "step": 1},

[0063] { "type": "eq", "value": 0, "index": 203},

[0064] { "type": "lt", "value": 0, "index": 204} ]

[0066] }

[0067] The hierarchical bucket configuration information in the JSON file can be considered as the configuration information of the bucket layer corresponding to the sort key of the power numerical attribute. Specifically, if this bucket layer is set as a leaf bucket layer, then the configuration information of the leaf bucket layer includes: the total number of buckets (bucket_count) of the leaf bucket layer is 205, and the bucket index range of the leaf bucket layer is 0-204. The unlimited constant (unlimited_value) is used to indicate that the value of the sort key corresponding to the leaf bucket can be infinite. The data in all leaf buckets of the leaf bucket layer is sorted in ascending order ("order": "asc"), and the bucket index rule array (rules[]) represents the following meanings:

[0068] Unlimited indicates that the value is of the infinite rule type, and if the sort key value is of the infinite rule type, then its corresponding bucket index in the leaf bucket layer is 0;

[0069] Gte represents a rule type where the value is greater than or equal to the threshold. If the value of the sort key belongs to the rule type greater than or equal to 1000000000, then its corresponding bucket index in the leaf bucket layer is 1.

[0070] Range represents the rule type within a closed interval (i.e., the range of values). If the value of the sort key belongs to the rule type within the closed intervals of 201 and 999999999, then its corresponding bucket index in the leaf bucket layer is 2.

[0071] `range_to_index` represents a range-by-step mapping (i.e., the relationship between the value range and the index) rule type. If the value of the sort key belongs to the range of 1 to 200, the corresponding bucket index in the leaf bucket layer is 3 to 202, and the step size of the bucket index is 1. That is, the bucket index corresponding to the value N of the sort key (N represents any integer from 1 to 200) is `index_base + (N - min) = 3 + N - 1 = N + 2`. This rule type is beneficial for conveniently implementing finer-grained bucket control.

[0072] eq represents a rule type where the value is equal to a specific value. If the value of the sort key belongs to the rule type where the value is equal to a specific value (i.e., 0), then its corresponding bucket index in the leaf bucket layer is 203.

[0073] 1t indicates a rule type where the value is less than the threshold. If the value of the sort key belongs to the rule type where the value is less than the threshold (i.e., 0), then its corresponding bucket index in the leaf bucket layer is 204.

[0074] S101. Construct a bucket index mapper for each bucket layer according to the bucketing strategy information of each bucket layer.

[0075] In the multidimensional ordered data disclosed herein, each sort key corresponds to a bucket index mapper. When the number of buckets in the hierarchical bucket is the same as the number of sort keys in the multidimensional ordered data, each bucket in the hierarchical bucket corresponds to a sort key and a bucket index mapper.

[0076] The process of constructing the bucket index mapper disclosed herein can be considered as transforming bucketing strategy information into executable calculation logic from the sort key value to the bucket index; that is, it can be considered as the process of generating the mapping logic between the sort key value and the bucket index. For a specific example, for a sort key of a numeric attribute, the process of constructing the bucket index mapper may include: generating linear mapping logic or range search logic based on the sort key value / range of the numeric attribute, the number of buckets in the current bucket layer, and the bucket index of the current bucket layer. For another specific example, for a sort key of a discrete label attribute, the process of constructing the bucket index mapper may include: generating mapping logic from discrete labels to bucket indices based on a discrete label mapping table or hash rules. The bucket index mapper successfully constructed by this disclosure is used to calculate the bucket index corresponding to a sort key value. For example, the bucket index mapper can be in the form of a function. When the bucket index mapper is called, it receives a sort key value, calculates the bucket index corresponding to the sort key value according to preset logic, and returns the calculated bucket index.

[0077] In one example, this disclosure can verify the bucketing strategy information of each bucket layer before, during, or after constructing the bucket index mapper, so as to promptly detect whether there are any anomalies in the hierarchical bucket configuration information in the hierarchical bucket configuration file. For example, it can calculate whether the sum of the total number of buckets in each bucket layer is equal to the total number of buckets in the hierarchical bucket. Or, for any bucket layer, it can calculate whether the total number of buckets in that bucket layer is the same as the number of bucket indexes obtained by that layer through mapping. If an anomaly is detected in the hierarchical bucket configuration information in the hierarchical bucket configuration file, corresponding prompt information can be output.

[0078] In one example, the bucket index mappers constructed separately for each bucket level in this disclosure can be stored in a global mapping object array, and the contents of each element in the global mapping object array can be used to construct a hierarchical bucket index tree. The process of constructing the bucket index mappers separately for each bucket level in this disclosure can be set during the initialization process of the global mapping object array, and an example of the initialization process of the global mapping object array can be represented by the following pseudocode:

[0079] Algorithm InitMultiLayerMappers / / Algorithm for initializing a global array of mappers objects across multiple layers of buckets.

[0080] Input: config_path / / The input is the storage path of the hierarchical bucket configuration file

[0081] Output: mappers[] / / The output is a global array of mappers objects, where each element corresponds one-to-one with a bucket level in the hierarchical bucket system.

[0082] text = ReadFile(json_path) / / Reads all text content from the hierarchical bucket configuration file at the specified path.

[0083] root = ParseConfig(text) / / Parse all text content to generate the root object root.

[0084] compare = root.compare(optional) / / Extracts the configuration information of the global comparator in the root object, such as whether the data is sorted in ascending or descending order.

[0085] For each layer in root.layers: iterate through each bucket layer in the root object.

[0086] cfg.precision = layer.precision (optional) / / Extracts the numerical precision configuration information of the current bucket layer (optional), used to unify precision during bucket index mapper calculation.

[0087] cfg.bucket_count = layer.bucket_count (optional) / / Retrieves the total number of buckets in the current bucket layer

[0088] cfg.unlimited_value = layer.unlimited_value (optional) / / Extract the infinite constant of the current bucket layer

[0089] cfg.default_index = layer.default_index (optional) / / Extract the index of the bottom bucket of the current bucket layer

[0090] for rule in layer.rules: / / Iterate through all bucketing rules in the current bucket layer

[0091] `parse rule.type and fields` / / Parse the rule type and parameters (such as threshold, range, starting index, and index step size) of the bucketing rule being traversed.

[0092] cfg.rules.append(rule) / / Summarize the bucketing rules after parsing

[0093] mapper = BuildLayerMapper(cfg) / / Based on the complete bucketing strategy information cfg of this layer, instantiate and generate the mapping object mapper for the current bucket layer. This mapping object includes the bucket index mapper for the current bucket layer.

[0094] mappers.append(mapper) / / Adds the generated map objects for the current bucket to the global map object array mappers.

[0095] return mappers / / Returns a global array of mappers objects.

[0096] As can be seen from the pseudocode above, each element mapper in the global mapping object array mappers can include not only the bucket index mapper of the corresponding bucket level, but also other content, such as the content required to build a hierarchical bucket index tree. Specifically, any element mappers[i] in the global mapping object array mappers can also include: the numerical precision configuration information of bucket level i, the total number of buckets of bucket level i, the infinite constant of bucket level i, and the catch bucket index of bucket level i, etc. In addition, for the sorting key of numerical attributes, the corresponding element mappers in the global mapping object array mappers can also include the global comparator compare.

[0097] S102. Based on the bucketing strategy information of each bucket layer, set the number of bucket nodes in each bucket layer of the hierarchical bucket index tree, and assign bucket indexes to each bucket node in each bucket layer to generate the hierarchical bucket index tree.

[0098] The root node of the hierarchical bucket index tree in this disclosure is the top-level node of the hierarchical bucket index tree. That is, the bucket nodes of each bucket level in this disclosure are attached to the root node, and the number of bucket nodes attached to the root node is usually the same as the number of sort keys. For example, for the multidimensional ordered data (region, sales amount), the root node of the hierarchical bucket index tree has two levels of bucket nodes. The first level of bucket nodes corresponds to the region, and the second level of bucket nodes, i.e., the leaf bucket nodes, corresponds to the sales amount.

[0099] In one example, this disclosure can generate a hierarchical bucket index tree by constructing bucket nodes layer by layer from the root node downwards. For example, after setting the root node of the hierarchical bucket index tree, the number of child nodes of the root node is set according to the total number of buckets in the next bucket layer of the root node (such as cfg.bucket_count in mappers[0]), and bucket indexes (including bottom bucket indexes) are set for each child node of the root node according to the bucketing rules of the bucket layer (such as cfg.rules in mappers[0]). Then, for each child node of the root node, the child nodes are deployed (i.e., the child nodes of the child nodes of the root node are deployed) until all leaf bucket nodes are deployed, thereby forming a hierarchical bucket index tree.

[0100] In one example, the hierarchical bucket index tree constructed in this step can be a full hierarchical bucket index tree. A full hierarchical bucket index tree refers to a hierarchical bucket index tree where all bucket levels are full bucket levels, and the full hierarchical bucket index tree can be considered as the full index skeleton. In this disclosure, a full bucket level refers to a bucket level where the number of bucket nodes set for that level is the same as the total number of buckets in the bucketing strategy information for that level. Specifically, during the process of generating the hierarchical bucket index tree level by level, whether it is a bucket level corresponding to the sort key of a numerical attribute or a bucket level corresponding to the sort key of a discrete label attribute, the number of bucket nodes in that level is set to the total number of buckets in the bucketing strategy information for that level, and a bucket index is assigned to each bucket node in that level (including the catch bucket node), thereby generating a full bucket level corresponding to that sort key. After successfully constructing each full bucket level level, a full hierarchical bucket index tree is formed. When the number of bucket nodes in each bucket layer is not large, the structure of the hierarchical bucket index tree is fixed and complete by constructing a full hierarchical bucket index tree, which helps to simplify the data addressing logic. As a result, the time complexity of the addressing process for operations such as insertion and deletion of multidimensional ordered data is O(1), which in turn helps to improve data read and write performance.

[0101] In another example, the hierarchical bucket index tree constructed in this step can also be a sparse hierarchical bucket index tree. A sparse hierarchical bucket index tree is a hierarchical bucket index tree in which at least one bucket layer is a sparse bucket layer, and the sparse hierarchical bucket index tree can be considered as a sparse index skeleton. In this disclosure, a sparse bucket layer refers to a bucket layer whose number of bucket nodes is less than the total number of buckets in the bucketing strategy information of that bucket layer. Specifically, in the process of generating the hierarchical bucket index tree bucket by bucket layer, whether it is a bucket layer corresponding to a sorting key of a numerical attribute or a bucket layer corresponding to a sorting key of a discrete label attribute, the number of bucket nodes in that bucket layer can be set to a value less than the total number of buckets in the bucketing strategy information of that bucket layer, such as setting it to 1 / n (where n is an integer greater than 1) of the total number of buckets in the bucketing strategy information of that bucket layer. Then, a bucket index is assigned to each bucket node (including the bottom bucket node) in that bucket layer, thereby generating the sparse bucket layer corresponding to the sorting key. After successfully constructing the corresponding sparse bucket layer or full bucket layer layer by layer, a sparse hierarchical bucket index tree is formed. Furthermore, when assigning bucket indexes to bucket nodes, bucket indexes can be assigned based on the frequency of occurrence (i.e., commonality) and / or the uniformity of value distribution of the sort key. For example, bucket nodes can be prioritized for buckets corresponding to high-frequency values ​​and for buckets corresponding to values ​​with high distribution density. This helps improve the hit rate (i.e., success rate) of data location. Moreover, this disclosure allows setting commonality parameters and some density parameters for bucket indexes in the hierarchical bucket configuration information in the hierarchical bucket configuration file. This allows determining the bucket indexes of the initially set bucket nodes in the sparse bucket layer based on the bucketing strategy information of each bucket layer. Additionally, when the sort key for a numerical attribute has a very large number of different values, this disclosure should set a sparse bucket layer for the bucket layer corresponding to that sort key. Similarly, when the sort key for a discrete label attribute has a very large number of different values ​​(i.e., many categories), this disclosure can also set a sparse bucket layer for the bucket layer corresponding to that sort key.

[0102] This disclosure avoids the phenomenon that the structure of the hierarchical bucket index tree is too large in the initial stage by constructing a sparse hierarchical bucket index tree. This helps to avoid the problem of the hierarchical bucket index tree consuming system resources in a concentrated manner during the initialization process, which leads to excessive system initialization time. In other words, it helps to improve the initialization speed of the hierarchical bucket index tree.

[0103] In one example, this disclosure can construct a hierarchical bucket index tree by recursively calling the node construction function layer by layer. That is, this disclosure pre-sets a node construction function, which is used to construct all child nodes of any node. Thus, this disclosure constructs the child nodes of a node sequentially from the root node down until the construction of all leaf bucket nodes in the leaf bucket layer is completed.

[0104] The process of initializing the hierarchical bucket index tree (i.e., the full hierarchical bucket index tree) can be represented in the form of the following pseudocode:

[0105] Algorithm InitBucketTree / / Algorithm used to initialize a hierarchical bucket index tree

[0106] Input: mappers[] / / Input: a global array of mappers objects

[0107] Output: root / / Output: The root node of the successfully constructed hierarchical bucket index tree.

[0108] root = NewBucketNode() / / Creates the root node object of the hierarchical bucket index tree. This root node object is the starting entry point of the entire hierarchical bucket index tree.

[0109] function Build(level, node): / / Defines the node building function, where level represents the bucket level currently being built, and node represents the parent node of the child node to be created;

[0110] count = mappers[level].bucket_count / / Read the total number of buckets in the currently being built bucket level from the global array of mappers objects to determine the total number of child nodes of the current parent node.

[0111] node.children = Allocate(count) / / Allocate memory in batches according to the total number of child nodes, create count empty bucket nodes, and store each empty bucket node in the child node array of the current parent node.

[0112] if level + 1 < L: / / If the leaf bucket level has not been reached, L represents the total number of layers in the hierarchical buckets.

[0113] for each child in node.children: / / Iterate through the array of child nodes of the current parent node.

[0114] Build(level + 1, child) / / Recursively calls the Build function to build the next bucket of nodes for the current parent node.

[0115] Build(0, root) / / Executes the node construction function to build a hierarchical bucket index tree layer by layer, starting from level 0 (the level where the root node is located).

[0116] return root / / Returns the root node of the successfully constructed hierarchical bucket index tree.

[0117] In one example, the multidimensional ordered data disclosed herein is defined as (direction of force, power), where the direction of force is the sort key of the discrete label attribute, and the power is the sort key of the numerical attribute. An example of the hierarchical bucket index tree built by this disclosure for this multidimensional ordered data is as follows: Figure 2 As shown.

[0118] Figure 2 The hierarchical bucket index tree shown consists of three levels, with the first level being the root node ( Figure 2 (Not shown in the diagram), the second layer consists of child nodes of the root node, and the third layer consists of leaf bucket nodes. The second layer corresponds to the direction of stress, and the third layer corresponds to power. The force direction in the disclosure includes three types. Therefore, this disclosure sets up a bucket node for each direction. The bucket indices of these three bucket nodes are 0, 1, and 2, respectively. That is, the three bucket nodes can be represented as sidebucket[0], sidebucket[1], and sidebucket[2], and each of these three bucket nodes has at least one leaf bucket node. Figure 2 Only the leaf bucket nodes of sidebucket[1] are shown schematically. The bucket indices of all leaf bucket nodes of sidebucket[1] are 0, 1, 2, 3, ... 203 and 204, respectively. That is, each leaf bucket node can be represented as: bucket[0], bucket[1], bucket[2], bucket[3], bucket

[203] and bucket

[204] . Each leaf bucket node is provided with a doubly linked list, so that all multidimensional ordered data are distributed in the doubly linked list of the corresponding leaf bucket node according to the direction of force and the magnitude of power.

[0119] S103. Set up storage containers for each leaf bucket node in the hierarchical bucket index tree.

[0120] Each storage container in this disclosure corresponds to a leaf bucket node, and different storage containers correspond to different leaf bucket nodes. The storage containers can be considered to be set inside the leaf bucket nodes. Therefore, the storage containers in this disclosure are mainly used to store multidimensional ordered data belonging to the leaf bucket node.

[0121] In one example, regardless of whether the sorting key corresponding to the leaf bucket layer is a numerical attribute sorting key or a discrete label attribute sorting key, the storage container set for the distribution of each leaf bucket node in this disclosure can be a doubly linked list (such as a doubly linked list with head and tail sentinels or a doubly linked list without head and tail sentinels, etc.). Of course, it can also be a dynamic array, a skip list, or a balanced binary tree, or other forms of storage container. This disclosure does not limit the specific form of the storage container for the leaf bucket node. By using a doubly linked list as a storage container, not only can the time complexity of insertion and deletion operations of multidimensional ordered data in the doubly linked list be O(1) when the location of the multidimensional ordered data is known, but it is also beneficial to use a bidirectional positioning method to locate the position of multidimensional ordered data in the doubly linked list, thereby further improving the access efficiency of multidimensional ordered data.

[0122] S104. For the multidimensional ordered data to be added, based on the values ​​of each sort key in the multidimensional ordered data to be added, the bucket index mapper of each bucket layer, and the hierarchical bucket index tree, determine the leaf bucket node to which the multidimensional ordered data to be added belongs, and store the multidimensional ordered data to be added in the storage container of its leaf bucket node.

[0123] In one example, this disclosure can obtain the values ​​of each sort key in the multidimensional ordered data to be added dimension by dimension. For example, the values ​​of each sort key can be obtained according to the order of the fields. Then, the values ​​of each sort key are provided to the bucket index mapper of the corresponding bucket layer. Based on the output of each bucket index mapper, the bucket index corresponding to each sort key value can be obtained. In this way, the corresponding bucket node can be located layer by layer in the hierarchical bucket index tree using these bucket indexes. The location range can be gradually narrowed down from the root node of the hierarchical bucket index tree until a leaf bucket node in the leaf bucket layer is located, thereby obtaining the leaf bucket node to which the multidimensional ordered data to be added belongs. It should be noted that if the bucket index mapper of the corresponding bucket layer cannot provide a corresponding bucket index for the input sort key value, this disclosure can use the fallback bucket index as the bucket index corresponding to the sort key value and continue to locate the bucket node for the downstream bucket layer.

[0124] In one example, an example of the operation of a bucket index mapper configured in Table 1 above is shown below. Figure 3 As shown.

[0125] Figure 3The bucket index mapper internally contains a rule list, which includes multiple mapping rules, such as those for the `unlimited` rule type, `eq` rule type, `1t` rule type, `lte` rule type, `gt` rule type, `gte` rule type, `range` rule type, and `range_to_index` rule type. If the bucket index mapper receives an input (key) of 0, it will match the rules sequentially and output bucket index 203 using the matched `eq` rule. If the bucket index mapper receives an input (key) of 150, it will match the rules sequentially and output bucket index 152 using the matched `range_to_index` rule.

[0126] In one example, for a bucket layer, if the bucket index mapper corresponding to that bucket layer is a quantized bucket index mapper, then when locating leaf bucket nodes for newly added multidimensional ordered data and obtaining the bucket index corresponding to the sort key value using the bucket index mapper of that bucket layer, the sort key value should first be quantized (e.g., multiplied by a predetermined factor and then rounded or rounded up / down, etc.). Then, the quantized value is provided as input to the bucket index mapper, and the output value of the bucket index mapper is used as the bucket index corresponding to the sort key value. It should be noted that if the bucket index mapper of that bucket layer cannot provide a corresponding bucket index for the quantized input value, this disclosure can also use the fallback bucket index as the bucket index corresponding to the sort key value and continue locating bucket nodes for downstream bucket layers.

[0127] In one example, if the sort key in multidimensional ordered data is set to a floating-point type, the process of quantizing floating-point data can be represented by the following pseudocode:

[0128] Algorithm QuantizeFloat / / Quantization algorithm for floating-point numbers

[0129] Input: value (float), precision / / Input is the floating-point number to be quantized and the quantization precision.

[0130] Output: key_int / / Output is the quantized integer

[0131] scale = 1 / precision / / Convert quantization precision to magnification factor, i.e., magnify by 100 times.

[0132] key_int = round(value * scale) # or floor / ceil / / Magnify the floating-point number by 100 and then round it down or up.

[0133] return key_int / / Returns the quantized integer.

[0134] If the quantization precision = 0.01 and the value of a sort key is 12.34, then the quantized integer key_int obtained using the pseudocode above will be 1234. Subsequently, this disclosure can provide 1234 as input to the bucket index mapper to obtain the corresponding bucket index. Quantizing the sort key value using a pre-set precision helps ensure the accuracy of the bucket index corresponding to the floating-point sort key value.

[0135] In one example, during the process of building a hierarchical bucket index tree (such as the initialization process), if the hierarchical bucket index tree is a sparse hierarchical bucket index tree, then for the sparse bucket layer in the index tree, although the bucket index mapper corresponding to the sparse bucket layer can provide the corresponding bucket index for the value of the corresponding sort key, the bucket index may not have a corresponding bucket node in the sparse bucket layer. This disclosure can dynamically create the missing corresponding bucket node during the data insertion process, thereby continuously improving the sparse hierarchical bucket index tree during the data insertion process, and eventually potentially forming a full hierarchical bucket index tree. This disclosure dynamically creates missing bucket nodes in the sparse bucket layer. On the one hand, it avoids the storage space waste and initialization delay caused by pre-allocating all bucket nodes during the initialization phase of the hierarchical bucket index tree. On the other hand, as data is continuously inserted, the bucket nodes are gradually filled in. Without additional reconstruction of the hierarchical bucket index tree, the sparse hierarchical bucket index tree can smoothly evolve into a full hierarchical bucket index tree, ensuring the accurate location of leaf bucket nodes. Thus, it achieves lightweight startup of the hierarchical bucket index tree while taking into account storage space utilization and positioning performance, making it particularly suitable for application scenarios with large amounts of data and sparse data distribution.

[0136] In one example, the storage container allocated to each leaf bucket node in the hierarchical bucket index tree in this disclosure is a doubly linked list. When the leaf bucket node to which the multidimensional ordered data to be added belongs is located, the insertion position of the multidimensional ordered data to be added can be found starting from the head node of the doubly linked list of the leaf bucket node. For example, when the doubly linked list is not empty and the multidimensional ordered data only includes a sort key (ascending order) of a numerical attribute, this disclosure can sequentially compare the value of the corresponding sort key in the doubly linked list of the located leaf bucket node from the head node to the tail node to see if it is greater than (or greater than or equal to) the value of the corresponding sort key of the multidimensional ordered data to be added. When the first node that is greater than (or greater than or equal to) is found, the multidimensional ordered data to be added is inserted before that node. If there is no node that is greater than, the multidimensional ordered data to be added is inserted at the tail. For example, when the doubly linked list is not empty and the multidimensional ordered data includes sorting keys (ascending order) for multiple numerical attributes, the process of finding the insertion position of the new multidimensional ordered data in the doubly linked list of the located leaf bucket node and inserting it into the doubly linked list can be as follows:

[0137] Step 1-1: Set the first data node of the doubly linked list as the current data node;

[0138] Step 1-2: Determine whether the value of the corresponding sort key of the data in the current data node is greater than the value of the sort key of the first numerical attribute in the multidimensional ordered data to be added.

[0139] If it is greater than, then the data node formed by the new multidimensional ordered data is inserted into the doubly linked list as the upstream node of the current data node, and the process of finding the insertion position and inserting ends.

[0140] If it is less than, then the downstream node of the current data node is taken as the current data node, and the above steps 1-2 are returned;

[0141] If it equals, proceed to steps 1-3;

[0142] Steps 1-3: Continue to determine whether the sort key of the second numerical attribute in the current data node is greater than the sort key of the second numerical attribute in the new multidimensional ordered data. If it is greater, insert the data node formed by the new multidimensional ordered data as the upstream node of the current data node into the doubly linked list, and the process of finding the insertion position and inserting ends. If it is less than, take the downstream node of the current data node as the current data node and return to Steps 1-2 above. If it is equal, continue to determine whether the sort key of the third numerical attribute in the current data node is greater than the sort key of the third numerical attribute in the new multidimensional ordered data. Continue in this manner until the new multidimensional ordered data is inserted into the doubly linked list.

[0143] In one example, the storage container allocated to each leaf bucket node in the hierarchical bucket index tree in this disclosure is a doubly linked list. When the leaf bucket node to which the multidimensional ordered data to be added belongs is located, the insertion position of the multidimensional ordered data to be added can be found starting from the tail node of the doubly linked list of the leaf bucket node. For example, when the doubly linked list is not empty and the multidimensional ordered data only includes a sort key (ascending order) of a numerical attribute, this disclosure can sequentially compare the value of the corresponding sort key in the doubly linked list of the located leaf bucket node from the tail node to the head node to see if it is less than (or less than or equal to) the value of the corresponding sort key of the multidimensional ordered data to be added. When the first node less than (or less than or equal to) is found, the multidimensional ordered data to be added is inserted after that node. If there is no node less than, the multidimensional ordered data to be added is inserted at the head end. For example, when the doubly linked list is not empty and the multidimensional ordered data includes sorting keys (ascending order) for multiple numerical attributes, the process of finding the insertion position of the new multidimensional ordered data in the doubly linked list of the located leaf bucket node and inserting it into the doubly linked list can be as follows:

[0144] Step 2-1: Take the last data node of the doubly linked list as the current data node;

[0145] Step 2-2: Determine whether the sort key of the first numerical attribute of the data in the current data node is less than the sort key of the first numerical attribute of the multidimensional ordered data to be added.

[0146] If it is less than, then the data node formed by the new multidimensional ordered data is inserted into the doubly linked list as the downstream node of the current data node, and the process of finding the insertion position and inserting ends.

[0147] If it is greater than, then the upstream node of the current data node is taken as the current data node, and the process returns to step 2-2 above.

[0148] If it equals, proceed to step 2-3;

[0149] Step 2-3: Continue to determine whether the sort key of the second numerical attribute of the data in the current data node is less than the sort key of the second numerical attribute of the data to be added in the multidimensional ordered data. If it is less, insert the data node formed by the data to be added in the multidimensional ordered data as the downstream node of the current data node into the doubly linked list. The process of finding the insertion position and inserting ends here. If it is greater, take the upstream node of the current data node as the current data node and return to step 2-2 above. If it is equal, continue to determine whether the sort key of the third numerical attribute of the data in the current data node is less than the sort key of the third numerical attribute of the data to be added in the multidimensional ordered data. Continue in this manner until the data to be added in the multidimensional ordered data is inserted into the doubly linked list.

[0150] In one example, the storage container allocated to each leaf bucket node in the hierarchical bucket index tree in this disclosure is a doubly linked list. When the leaf bucket node to which the newly added multidimensional ordered data belongs is located, the insertion position of the newly added multidimensional ordered data can be searched simultaneously from the head and tail nodes of the doubly linked list of that leaf bucket node, i.e., a bidirectional parallel search for the insertion position. For example, if the doubly linked list is not empty and the multidimensional ordered data only includes a sorting key (ascending order) of a numerical attribute, this disclosure can sequentially compare the value of the corresponding sorting key in the doubly linked list of the located leaf bucket node from the head node to the tail node to see if it is greater than (or...). The disclosure retrieves the value of the corresponding sort key of the multidimensional ordered data to be added (greater than or equal to). Simultaneously, in the doubly linked list of the located leaf bucket node, the disclosure sequentially compares the value of the corresponding sort key in each node from the tail node to the head node to see if it is less than (or less than or equal to) the value of the corresponding sort key of the multidimensional ordered data to be added. When the first node greater than (or greater than or equal to) is found in the backward direction or the first node less than (or less than or equal to) is found in the forward direction, the multidimensional ordered data to be added is inserted into the doubly linked list. If the searches in the two directions meet / collide, the multidimensional ordered data to be added is inserted at the meeting / collision position. For example, if the doubly linked list is not empty and the multidimensional ordered data includes sort keys (ascending order) for multiple numerical attributes, the disclosure can perform the following process: (The process of searching for the insertion position of the multidimensional ordered data to be added in parallel in both directions within the doubly linked list of the located leaf bucket node and inserting it into the doubly linked list.)

[0151] The process of processing from the head node onwards:

[0152] Step 3-11: Take the first data node of the doubly linked list as the first current data node;

[0153] Step 3-12: Determine whether the value of the corresponding sort key of the data in the current data node is greater than the value of the sort key of the first numerical attribute in the multidimensional ordered data to be added.

[0154] If it is greater than, then the data node formed by the new multidimensional ordered data will be inserted into the doubly linked list as the upstream node of the current data node. The process of finding the insertion position and inserting ends, that is, the process of processing from the head node backward and the process of processing from the tail node forward both end.

[0155] If it is less than, then the downstream node of the current data node is taken as the current data node, and the process returns to steps 3-12 above.

[0156] If it equals, proceed to step 3-13;

[0157] Step 3-13: Continue to determine whether the sort key of the second numerical attribute in the current data node is greater than the sort key of the second numerical attribute in the new multidimensional ordered data. If it is greater, insert the data node formed by the new multidimensional ordered data as the upstream node of the current data node into the doubly linked list. This process of finding the insertion position and inserting ends, that is, both the process of processing from the head node backwards and the process of processing from the tail node forwards end. If it is less than, take the downstream node of the current data node as the current data node and return to step 3-12 above. If it is equal, continue to determine whether the sort key of the third numerical attribute in the current data node is greater than the sort key of the third numerical attribute in the new multidimensional ordered data. Continue in this manner until the new multidimensional ordered data is inserted into the doubly linked list or the process of processing from the tail node forwards and backwards is encountered.

[0158] The process of processing from the tail node forward:

[0159] Step 3-21: Take the last data node of the doubly linked list as the current data node;

[0160] Step 3-22: Determine whether the sort key of the first numerical attribute of the data in the current data node is less than the sort key of the first numerical attribute of the multidimensional ordered data to be added.

[0161] If it is less than, then the data node formed by the new multidimensional ordered data will be inserted into the doubly linked list as the downstream node of the current data node. The process of finding the insertion position and inserting ends, that is, the process of processing from the head node backward and the process of processing from the tail node forward both end.

[0162] If it is greater than, then the upstream node of the current data node is taken as the current data node, and the process returns to step 3-22 above.

[0163] If it equals, proceed to step 3-23;

[0164] Step 3-23: Continue to determine whether the sort key of the second numerical attribute of the data in the current data node is less than the sort key of the second numerical attribute of the data to be added in the multidimensional ordered data. If it is less, insert the data node formed by the data to be added in the multidimensional ordered data as the downstream node of the current data node into the doubly linked list. The process of finding the insertion position and inserting ends, that is, the process of processing from the head node backward and the process of processing from the tail node forward both end. If it is greater, take the upstream node of the current data node as the current data node and return to step 3-22 above. If it is equal, continue to determine whether the sort key of the third numerical attribute of the data in the current data node is less than the sort key of the third numerical attribute of the data to be added in the multidimensional ordered data. And so on, until the data to be added in the multidimensional ordered data is inserted into the doubly linked list or the process of processing from the tail node forward and backward is encountered.

[0165] The collision between the above-mentioned process of processing from the head node backward and the process of processing from the tail node forward can be considered as follows: the first current data node and the second current data node are adjacent. That is, if the second current data node needs to be used as the first current data node or the first current data node needs to be used as the second current data node during the process of processing from the head node backward, the two processing directions can be considered to collide.

[0166] In one example, if the process of processing backwards from the head node collides with the process of processing forwards from the tail node, this disclosure can insert the data node to be formed from the new multidimensional ordered data at the collision position, that is, between the first current data node and the second current data node, so that the data node to be formed from the new multidimensional ordered data becomes the downstream node of the first current data node and the upstream node of the second current data node.

[0167] In one example, when the doubly linked list is empty, this disclosure allows directly inserting the data node formed by the new multidimensional ordered data as the first data node into the doubly linked list. Furthermore, examples of data sorted in descending order can be found in the description of the above embodiments, and will not be detailed here.

[0168] In one example, after determining the insertion position of the data node formed by the new multidimensional ordered data in the doubly linked list, this disclosure inserts the data node into the doubly linked list by setting the predecessor and successor pointers of the data node, and modifying the successor pointer of its upstream node (if any) and the predecessor pointer of its downstream node (if any). For example, Figure 4The doubly linked list shown is a doubly linked list with head and tail sentinels. This doubly linked list stores three data nodes: data node n1, data node n2, and data node n3. This disclosure sets a left pointer that moves backward from the head node (head) and a right pointer that moves forward from the tail node (tail). The data node pointed to by the left pointer is the first current data node mentioned above, and the data node pointed to by the right pointer is the second current data node mentioned above. It is set that during the backward movement of the left pointer, the insertion position of the new multidimensional ordered data (e.g., ...) is obtained. Figure 4 The position indicated by the dashed arrow (between data node n1 and data node n2) allows this disclosure to modify the successor pointer of data node n1 and the predecessor pointer of data node n2, making both the successor pointer of data node n1 and the predecessor pointer of data node n2 point to the new data node (node_new), and making the predecessor pointer of the new data node point to data node n1, and the successor pointer of the new data node point to data node n2. This enables the new data node to be successfully inserted into the doubly linked list, forming a structure as shown below. Figure 5 The doubly linked list shown.

[0169] By searching for the insertion position of the new multidimensional ordered data from the head and tail nodes of the doubly linked list of the leaf bucket nodes towards the middle of the doubly linked list, the unidirectional linear scan in the doubly linked list is optimized into a bidirectional parallel search, which helps to improve the data insertion and positioning efficiency, and further improves the processing efficiency of multidimensional ordered data.

[0170] In one example, where the last sort key in the multidimensional ordered data is a numerical attribute sort key, and the other sort keys are discrete label attributes sort keys, the process of inserting the new multidimensional ordered data into the doubly linked list using bidirectional parallel search can be represented in the following pseudocode form:

[0171] Algorithm InsertWithBuckets / / Insertion algorithm

[0172] Input: node / / Input: The node to be inserted

[0173] Output: none / / Output: No return value

[0174] for level in [0..L-1]: / / Traverse each level in the hierarchical bucket index tree, where L is the level number.

[0175] key[level] = KeyProvider(node, level) / / Extract the value of the sort key for the current level.

[0176] index[level] = Mapper[level].Map(key[level]) / / Use this value to obtain the bucket index

[0177] b = LeafBucket(index[0..L-1]) / / Locate the leaf bucket node using the bucket index of each level

[0178] key_last = key[L-1] / / Extract the value of the sort key corresponding to the leaf bucket layer

[0179] if b.head == null: / / The doubly linked list inside the leaf bucket is empty.

[0180] b.head = node; b.tail = node; return / / Set the nodes to be inserted as the head and tail nodes.

[0181] if key_last <= b.head.key: / / Head boundary insertion processing

[0182] InsertBefore(b, b.head, node); return / / Insert before the current head node

[0183] if key_last > b.tail.key: / / Tail boundary insertion processing

[0184] InsertAfter(b, b.tail, node); return / / Insert after the current tail node

[0185] left = b.head; right = b.tail / / Initialize two pointers

[0186] while left != null and right != null: / / The two pointers did not meet

[0187] if key_last <= left.key: / / Left pointer position

[0188] InsertBefore(b, left, node); return / / Insert before the node pointed to by the left pointer.

[0189] if key_last > right.key: / / Right pointer hit position

[0190] InsertAfter(b, right, node); return / / Insert after the node pointed to by the right pointer

[0191] If left == right or left.next == right: / / Two pointers collide

[0192] InsertAfter(b, left, node); return / / Insert between the two pointers

[0193] left = left.next / / Move the left pointer one step forward

[0194] right = right.prev / / Move the right pointer forward one step

[0195] Additionally, it should be noted that if the sorting key corresponding to the leaf bucket layer is the sorting key of the discrete label attribute (such as male / female), then after locating the leaf bucket node to which the multidimensional ordered data to be added belongs, this disclosure can directly add the multidimensional ordered data to be added to the tail of the doubly linked list of the leaf bucket node without performing a value comparison operation. Of course, it can also be added directly to the head of the doubly linked list.

[0196] In one example, when the multidimensional ordered data is actually two-dimensional ordered data, the structure of each node in the doubly linked list of this disclosure can be represented in the form of the following pseudocode:

[0197] struct Node { / / Structure of nodes in a doubly linked list

[0198] double a; / / One dimension in a two-dimensional ordered data set

[0199] double b; / / One of the other dimensions in two-dimensional ordered data

[0200] Node* prev; / / Pointer to the previous node

[0201] Node* next; / / Pointer to the successor of the node

[0202] }

[0203] Furthermore, the relationship between the sort key and the hierarchy of the hierarchical bucket index tree in this two-dimensional ordered data can be implemented using a pre-defined key provider, which is represented in the following pseudocode form:

[0204] struct KeyProvider { / / Definition of the key provider, which can provide corresponding sort keys based on the current layer.

[0205] return (level == 0) ? node.a : node.b; / / Level 0 corresponds to node 'a', and level 1 corresponds to node 'b'.

[0206] }

[0207] S105. For the multidimensional ordered data to be deleted, determine the leaf bucket node to which the multidimensional ordered data to be deleted belongs based on the values ​​of each sort key in the multidimensional ordered data to be deleted and the bucket index mapper of each bucket layer in the hierarchical bucket index tree, and delete the multidimensional ordered data to be deleted from the storage container of the leaf bucket node to which it belongs.

[0208] In one example, this disclosure can obtain the values ​​of each sort key in the multidimensional ordered data to be added dimension by dimension. For example, the values ​​of each sort key can be obtained according to the order of the fields. Then, the values ​​of each sort key are provided to the bucket index mapper of the corresponding bucket layer. Based on the output of each bucket index mapper, the bucket index corresponding to each sort key value can be obtained. In this way, the corresponding bucket node can be located layer by layer in the hierarchical bucket index tree using these bucket indexes. The location range can be gradually narrowed down from the root node of the hierarchical bucket index tree until a leaf bucket node in the leaf bucket layer is located, thereby obtaining the leaf bucket node to which the multidimensional ordered data to be deleted belongs. It should be noted that if the bucket index mapper of the corresponding bucket layer cannot provide a corresponding bucket index for the input sort key value, this disclosure can use the fallback bucket index as the bucket index corresponding to the sort key value and continue to locate the bucket node for the downstream bucket layer. After obtaining the leaf bucket node to which the multidimensional ordered data to be deleted belongs, the doubly linked list containing the multidimensional ordered data to be deleted can be determined. The upstream and downstream nodes of the multidimensional ordered data to be deleted can be obtained through the predecessor and successor pointers of the multidimensional ordered data to be deleted. By modifying the successor pointer of the upstream node and the predecessor pointer of the downstream node, and clearing the predecessor and successor pointers of the multidimensional ordered data to be deleted, the deletion operation of the multidimensional ordered data can be completed.

[0209] In one example, for a bucket layer, if the bucket index mapper corresponding to that bucket layer is a quantized bucket index mapper, then during the process of locating leaf bucket nodes for the multidimensional ordered data to be deleted and obtaining the bucket index corresponding to the sort key value using the bucket index mapper of that bucket layer, the sort key value should first be quantized (e.g., multiplied by a predetermined factor and then rounded or rounded up / down, etc.). Then, the quantized value is provided as input to the bucket index mapper, and the value output by the bucket index mapper is used as the bucket index corresponding to the sort key value. It should be noted that if the bucket index mapper of that bucket layer cannot provide a corresponding bucket index for the input quantized value, this disclosure can also use the fallback bucket index as the bucket index corresponding to the sort key value and continue to locate bucket nodes for downstream bucket layers.

[0210] In one example, the process of deleting multidimensional ordered data to be deleted from a doubly linked list can be represented by the following pseudocode:

[0211] Algorithm RemoveWithBuckets / / Removal algorithm

[0212] Input: node / / Input: the node to be deleted

[0213] Output: none / / Output: No return value

[0214] for level in [0..L-1]: / / Traverse each level in the hierarchical bucket index tree, where L is the level number.

[0215] key[level] = KeyProvider(node, level) / / Extract the value of the sort key for the current level.

[0216] index[level] = Mapper[level].Map(key[level]) / / Use this value to obtain the bucket index

[0217] b = LeafBucket(index[0..L-1]) / / Locate the leaf bucket node using the bucket index of each level

[0218] if node.prev != null: / / The node to be deleted is not the head node

[0219] node.prev.next = node.next / / Sets the successor pointer (next) of the upstream node of the node to be deleted to the downstream node of the node to be deleted.

[0220] else:

[0221] b.head = node.next / / Set the downstream node of the node to be deleted as the head node.

[0222] if node.next != null: / / The node to be deleted is not the tail node

[0223] node.next.prev = node.prev / / Sets the predecessor pointer (prev) of the downstream node to be deleted to the upstream node of the node to be deleted.

[0224] else:

[0225] b.tail = node.prev / / Use the upstream node of the node to be deleted as the head node

[0226] node.prev = null; node.next = null / / Disconnect the node to be deleted from the doubly linked list to prevent dangling pointers, accidental operations, etc.

[0227] In one example, when traversing multidimensional ordered data, this disclosure can traverse the multidimensional ordered data in the storage containers of all leaf bucket nodes according to the sorting of the bucket indices of all leaf bucket nodes. Furthermore, during the traversal, the current leaf bucket node can be checked to see if it is an empty bucket, i.e., whether its storage container is empty. If the result is that the current leaf bucket node is empty, it can be skipped, and the traversal can continue to the next leaf bucket node. If the result is that the current leaf bucket node is not empty, it can be traversed according to the storage order of the data in its storage container. When the storage container is a doubly linked list, this disclosure can determine whether a leaf bucket node is empty by checking the head pointer of the doubly linked list. For example, if head == nullptr, the leaf bucket node is considered empty; otherwise, it is considered non-empty. This disclosure effectively avoids invalid scans of storage containers by performing empty bucket checks during traversal, thereby improving traversal efficiency and further enhancing the processing efficiency of multidimensional ordered data.

[0228] In one example, when multidimensional ordered data is arranged in ascending order, the process of traversing multidimensional ordered data can be represented in the following pseudocode form:

[0229] Algorithm IterateBucketsForward / / Forward traversal algorithm, i.e., ascending order traversal algorithm

[0230] Input: none / / Input: None

[0231] Output: Sequence of nodes (ascending)

[0232] it = begin() / / Get the head node of the first non-empty bucket.

[0233] while it != end(): / / The iterator has not reached the end

[0234] visit(*it) / / Access a node, such as reading data or performing statistics.

[0235] ++it / / The iterator moves downstream to the next non-empty bucket.

[0236] Return / / Iteration complete

[0237] Algorithm IterateBucketsBackward / / Reverse traversal algorithm, i.e., descending order traversal algorithm

[0238] Input: none / / Input: None

[0239] Output: a sequence of nodes in descending order.

[0240] it = rbegin() / / Get the tail node of the first non-empty bucket.

[0241] while it != rend(): / / Iterator not reached the head

[0242] visit(*it) / / Access a node, such as reading data or performing statistics.

[0243] ++it / / The iterator moves upstream to the next non-empty bucket.

[0244] Return / / Iteration complete

[0245] In one example, the technical solution of this disclosure is compared with the technical solution of the prior art in terms of forward traversal, reverse traversal, and random access. The difference in time complexity between the two is shown in Table 3 below:

[0246] Table 3

[0247] Forward traversal O(n) O(m) + O(k) Skip the empty bucket Reverse traversal O(n) O(m) + O(k) Skip the empty bucket Random access O(n) O(1) + O(k) Quick location

[0248] In Table 3 above, n is the total number of nodes in the doubly linked list, m is the number of non-empty buckets, and k is the average number of nodes in a bucket. As can be seen from Table 3, the technical solution disclosed herein outperforms existing technologies in terms of time complexity, whether in forward traversal, reverse traversal, or random access.

[0249] In one example, this disclosure can encapsulate the operations performed in steps S100 to S103 above, such as encapsulating them in a separate initialization module. This module can hide the underlying logic such as parsing hierarchical bucket configuration information, constructing comparators, and instantiating ordered bucket sets, and provides a very simple interface, which is beneficial to improving the maintainability and usability of the system. The input parameters of the callable interface provided by this module can include the hierarchical bucket configuration file and the output of the key provider. The encapsulated initialization module can be represented in the form of the following pseudocode:

[0250] Algorithm BuildOrderedSet / / Algorithm for creating an ordered bucket set instance

[0251] Input: config_path, KeyProvider / / Input: The storage path of the hierarchical bucket configuration file and the key provider.

[0252] Output: OrderedSet / / Output: An instance of an ordered bucket set

[0253] settings = LoadMultiLevelSettings(json_path) / / Get the bucketing strategy information (including sorting direction) for each bucket level.

[0254] compare = BuildCompare(settings.compare) / / Generates the comparators within the leaf bucket nodes

[0255] set = OrderedMultiLevelSet(settings.layers, KeyProvider, compare) / / Creates an instance of an ordered bucket set.

[0256] return set / / Returns an instance of an ordered bucket set, which includes bucket index mappers for each level, hierarchical bucket index trees, and comparators, etc., for implementing operations such as insertion, deletion, and traversal of multidimensional ordered data.

[0257] In one example, this disclosure can encapsulate operations such as insertion, deletion, and traversal of multidimensional ordered data, for instance, within a separate module for using ordered bucket collection instances. By encapsulating these operations into a module, upper-level business code can manage multidimensional ordered data efficiently without needing to be aware of the underlying logic such as bucket node location, doubly linked list maintenance, and jumps across empty buckets. This improves the system's usability and maintainability. The encapsulated module for using ordered bucket collection instances can be represented in the following pseudocode form:

[0258] Algorithm UseOrderedSet / / Algorithm for using an instance of an ordered bucket set

[0259] Input: set, node / / Input: an instance of an ordered bucket set and the node to be operated on (e.g., a node to be added, a node to be deleted, etc.)

[0260] Output: none

[0261] `set.insert(node)` / / Inserts the data node into the hierarchical ordered bucket set instance; that is, the data node is inserted into the corresponding doubly linked list within the hierarchical ordered bucket set.

[0262] `set.remove(node)` / / Removes the data node from the hierarchical ordered bucket set instance, that is, removes the data node from the corresponding doubly linked list in the hierarchical ordered bucket set instance.

[0263] for each item in the set: / / Forward traversal (jumping across empty buckets)

[0264] visit(item) / / Access a data node, such as reading data or performing statistics.

[0265] for each item in reverse(set): / / Iterate in reverse (jumping across empty buckets)

[0266] visit(item) / / Access a data node, such as reading data or performing statistics.

[0267] Exemplary device

[0268] An embodiment of the apparatus disclosed herein for processing multidimensional ordered data using hierarchical bucketing is as follows: Figure 6 As shown. The apparatus of this embodiment can be used to implement the corresponding method embodiments of this disclosure. Figure 6The apparatus shown mainly includes: a parsing module 600, a mapper construction module 601, an index tree construction module 602, a container setting module 603, a data insertion module 604, and a data deletion module 605. Optionally, the apparatus may also include: a traversal module 606.

[0269] The parsing module 600 is mainly used to parse the hierarchical bucket configuration information in the hierarchical bucket configuration file and obtain the bucketing strategy information for each bucket layer based on the parsing results. The number of hierarchical bucket layers is determined by the number of sort keys in the multidimensional ordered data. In one example, the hierarchical bucket configuration information may include: global configuration information for all bucket layers and bucket layer configuration information for each bucket layer; when the sort keys in the multidimensional ordered data include sort keys of at least one numeric attribute, the global configuration information includes: the sorting method of all numeric attribute sort keys.

[0270] The mapper construction module 601 is mainly used to construct bucket index mappers for each bucket layer based on the bucketing strategy information obtained by the parsing module 600. Specifically, the bucket index mapper constructed for any bucket layer in this disclosure corresponds to a sorting key in multidimensional ordered data. Furthermore, when the layer bucket configuration information includes quantization processing rules for sorting keys of a predetermined data type, the bucket index mapper constructed by the mapper construction module 601 for bucket layers containing quantization processing rules is a quantized bucket index mapper.

[0271] The index tree building module 602 is mainly used to set the number of bucket nodes in each bucket layer in the hierarchical bucket index tree according to the bucketing strategy information of each bucket layer obtained by the parsing module 600, and to allocate bucket indexes to each bucket node in each bucket layer to generate the hierarchical bucket index tree.

[0272] In one example, during the process of generating a hierarchical bucket index tree bucket by bucket, the above-mentioned index tree building module 602 can set the number of full bucket nodes for the bucket layer corresponding to the sort key of the numerical attribute according to the total number of buckets in the bucketing strategy information of the bucket layer, and assign bucket indexes to all bucket nodes in the bucket layer respectively, generate the full bucket layer corresponding to the sort key, and finally form a full hierarchical bucket index tree.

[0273] In another example, during the process of generating a hierarchical bucket index tree bucket by bucket, the aforementioned index tree building module 602, for the bucket layer corresponding to the sort key of the numerical attribute, can set the number of sparse bucket nodes for that bucket layer based on the total number of buckets in the bucketing strategy information of that bucket layer, and assign bucket indexes to all bucket nodes in that bucket layer, generating a sparse bucket layer corresponding to that sort key. The number of sparse bucket nodes in this bucket layer is less than the total number of buckets in the bucketing strategy information of that bucket layer. The final hierarchical bucket index tree is a sparse hierarchical bucket index tree.

[0274] The container module 603 is primarily used to set up storage containers for each leaf bucket node in the hierarchical bucket index tree successfully constructed by the index tree construction module 602. For example, the container module 603 can set up storage containers in the form of doubly linked lists, dynamic arrays, skip lists, or balanced binary trees for each leaf bucket node in the hierarchical bucket index tree. Furthermore, when the leaf bucket node corresponds to a sorting key for a numerical attribute, the container module 603 can set up a doubly linked list for each leaf bucket node in the hierarchical bucket index tree.

[0275] The data insertion module 604 is mainly used to determine the leaf bucket node to which the new multidimensional ordered data belongs based on the values ​​of each sort key in the new multidimensional ordered data, the bucket index mapper of each bucket layer, and the hierarchical bucket index tree, and to store the new multidimensional ordered data in the storage container of its leaf bucket node.

[0276] In one example, for the sorting key corresponding to the sparse bucket layer in the multidimensional ordered data to be added, if the data insertion module 604 determines, based on the value of the sorting key and the bucket index mapper of the sparse bucket layer, that there is no bucket index corresponding to the value of the sorting key in the sparse bucket layer, the index tree construction module 602 can also add corresponding bucket nodes to the sparse bucket layer and its downstream bucket layers according to the bucketing strategy information corresponding to the sorting key, and assign bucket indexes to the added bucket nodes respectively. Then, the container setting module 603 can set storage containers for the added leaf bucket nodes respectively, so that the data insertion module 604 can store the multidimensional ordered data to be added in the storage container of its respective leaf bucket node.

[0277] In one example, the data insertion module 604 can search for the insertion position of the new multidimensional ordered data from the head and tail nodes of the doubly linked list of the leaf bucket nodes, respectively, towards the middle of the list. If the insertion position is successfully found in either direction, the data insertion module 604 stores the new multidimensional ordered data in the doubly linked list according to the found insertion position. If the search for the insertion position from the head node of the doubly linked list of the leaf bucket nodes towards the middle of the list meets, the data insertion module 604 can insert the new multidimensional ordered data at the meeting position in the doubly linked list.

[0278] In one example, when a sort key in the multidimensional ordered data to be added corresponds to a bucket index mapper based on quantization, the data insertion module 604 first quantizes the value of the sort key, and then uses the quantized value and the bucket index mapper based on quantization to determine the bucket index corresponding to the sort key.

[0279] The data deletion module 605 is mainly used to determine the leaf bucket node to which the multidimensional ordered data to be deleted belongs based on the values ​​of each sort key in the multidimensional ordered data to be deleted and the bucket index mapper of each bucket layer in the hierarchical bucket index tree, and delete the multidimensional ordered data to be deleted from the storage container of its leaf bucket node.

[0280] In one example, when a sort key in the multidimensional ordered data to be deleted corresponds to a bucket index mapper based on quantization, the data deletion module 605 first quantizes the value of the sort key, and then uses the quantized value and the bucket index mapper based on quantization to determine the bucket index corresponding to the sort key.

[0281] The traversal module 606 is mainly used to traverse the multidimensional ordered data in the storage container of each leaf bucket node according to the bucket index sorting of each leaf bucket node. During the traversal of the multidimensional ordered data, if it is determined that the storage container of any leaf bucket node is empty, the leaf bucket node is skipped.

[0282] The specific operations performed by each of the above modules can be found in the description of the above method embodiments, and will not be described in detail here.

[0283] Exemplary electronic devices

[0284] The following is for reference. Figure 7 To describe an electronic device according to embodiments of the present disclosure. Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Figure 7 As shown, the electronic device 71 includes one or more processors 711 and memory 712.

[0285] The processor 711 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 71 to perform desired functions.

[0286] The memory 712 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, and flash memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 711 may execute the program instructions to implement the methods for processing multidimensional ordered data using hierarchical bucketing as described in the various embodiments of this disclosure above, and / or other desired functions.

[0287] In one example, the electronic device 71 may further include an input device 713 and an output device 714, etc., these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown). Furthermore, the input device 713 may include, for example, a keyboard, a mouse, etc. The output device 714 can output various information to the outside. The output device 714 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0288] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device 71 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 71 may include any other suitable components depending on the specific application.

[0289] Exemplary computer program products and computer-readable storage media

[0290] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods for processing multidimensional ordered data using hierarchical buckets according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0291] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0292] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method for processing multidimensional ordered data using hierarchical buckets according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0293] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (not an exhaustive list) may include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0294] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0295] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0296] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0297] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0298] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0299] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0300] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for processing multidimensional ordered data using hierarchical bucketing, characterized in that, The method includes: The hierarchical bucket configuration information in the hierarchical bucket configuration file is parsed, and the bucketing strategy information of each bucket layer is obtained according to the parsing results; wherein, the number of hierarchical bucket layers is determined by the number of sorting keys in the multidimensional ordered data; A bucket index mapper is constructed for each bucket layer based on its respective bucketing strategy information; wherein, the bucket index mapper constructed for any bucket layer corresponds to a sorting key in the multidimensional ordered data; Based on the bucketing strategy information of each bucket layer, the number of bucket nodes in each bucket layer in the hierarchical bucket index tree is set, and a bucket index is assigned to each bucket node in each bucket layer to generate a hierarchical bucket index tree. Set up a storage container for each leaf bucket node in the hierarchical bucket index tree; For the new multidimensional ordered data, the leaf bucket node to which the new multidimensional ordered data belongs is determined based on the values ​​of each sorting key in the new multidimensional ordered data, the bucket index mapper of each bucket layer, and the hierarchical bucket index tree, and the new multidimensional ordered data is stored in the storage container of its leaf bucket node. For multidimensional ordered data to be deleted, the leaf bucket node to which the multidimensional ordered data to be deleted belongs is determined based on the values ​​of each sort key in the multidimensional ordered data to be deleted and the bucket index mapper of each bucket layer in the hierarchical bucket index tree, and the multidimensional ordered data to be deleted is deleted from the storage container of the leaf bucket node to which it belongs.

2. The method according to claim 1, characterized in that, The hierarchical bucket configuration information includes: global configuration information for all bucket layers and bucket layer configuration information for each bucket layer; When the sorting key in the multidimensional ordered data includes at least one numerical attribute sorting key, the global configuration information includes: the sorting method of all numerical attribute sorting keys.

3. The method according to claim 1, characterized in that, The step of setting the number of bucket nodes in each bucket layer of the hierarchical bucket index tree according to the bucketing strategy information of each bucket layer, and assigning bucket indexes to each bucket node in each bucket layer to generate the hierarchical bucket index tree includes: During the process of generating a hierarchical bucket index tree layer by layer, for the bucket layer corresponding to the sort key of the numerical attribute, the number of full bucket nodes of the bucket layer is set according to the total number of buckets in the bucketing strategy information of the bucket layer, and bucket indexes are assigned to all bucket nodes in the bucket layer to generate the full bucket layer corresponding to the sort key. or During the process of generating a hierarchical bucket index tree layer by layer, for the bucket layer corresponding to the sort key of the numerical attribute, the number of sparse bucket nodes in the bucket layer is set according to the total number of buckets in the bucketing strategy information of the bucket layer, and bucket indexes are assigned to all bucket nodes in the bucket layer to generate the sparse bucket layer corresponding to the sort key; wherein, the number of sparse bucket nodes in the bucket layer is less than the total number of buckets in the bucketing strategy information of the bucket layer.

4. The method according to claim 3, characterized in that, The method further includes: For the sorting key corresponding to the sparse bucket layer in the multidimensional ordered data to be added, if it is determined that there is no bucket index corresponding to the sorting key value in the sparse bucket layer based on the value of the sorting key and the bucket index mapper of the sparse bucket layer, then according to the bucketing strategy information corresponding to the sorting key, corresponding bucket nodes are added to the sparse bucket layer and each downstream bucket layer, bucket indexes are assigned to the added bucket nodes, and storage containers are set up for the added leaf bucket nodes. The multidimensional ordered data to be added is stored in the storage container of its respective leaf bucket node.

5. The method according to any one of claims 1 to 4, characterized in that, The first part describes the storage containers set up for each leaf bucket node in the hierarchical bucket index tree, including: For each leaf bucket node in the hierarchical bucket index tree, set a doubly linked list, a dynamic array, a skip list, or a balanced binary tree respectively.

6. The method according to claim 5, characterized in that, The first part describes the storage containers set up for each leaf bucket node in the hierarchical bucket index tree, including: Given the sorting key of the numerical attribute corresponding to the leaf bucket node, a doubly linked list is set up for each leaf bucket node in the hierarchical bucket index tree; The step of storing the new multidimensional ordered data in the storage container of its respective leaf bucket node includes: The insertion position of the new multidimensional ordered data is found from the head node and tail node of the doubly linked list of the leaf bucket node towards the middle of the list. If the insertion position of the new multidimensional ordered data is successfully found in either direction, the new multidimensional ordered data is stored in the doubly linked list according to the successfully found insertion position.

7. The method according to claim 6, characterized in that, The step of storing the new multidimensional ordered data in the storage container of its respective leaf bucket node also includes: If the search for an insertion position from the head node of the doubly linked list of the leaf bucket node towards the middle of the list meets with the search for an insertion position from the tail node of the doubly linked list of the leaf bucket node towards the middle of the list, the multidimensional ordered data to be added is inserted at the meeting position in the doubly linked list.

8. The method according to any one of claims 1 to 4, characterized in that, The method includes: Based on the bucket index sorting of each leaf bucket node, the multidimensional ordered data in the storage container of each leaf bucket node is traversed. During the traversal of the multidimensional ordered data, if it is determined that the storage container of any leaf bucket node is empty, the leaf bucket node is skipped.

9. The method according to any one of claims 1 to 4, characterized in that, The hierarchical bucket configuration information includes: quantization processing rules for sorting keys of a predetermined data type, and the bucket index mapper constructed for the bucket layer containing the quantization processing rules is a bucket index mapper based on quantization processing. Furthermore, determining the leaf bucket node to which the new multidimensional ordered data belongs based on the values ​​of each sort key in the data to be added, the bucket index mappers of each bucket layer, and the hierarchical bucket index tree includes: In the case where a sorting key in the multidimensional ordered data to be added corresponds to a bucket index mapper based on quantization, the value of the sorting key is quantized, and the bucket index corresponding to the sorting key is determined by using the quantized value and the bucket index mapper based on quantization. Furthermore, determining the leaf bucket node to which the new multidimensional ordered data belongs based on the values ​​of each sorting key in the data to be added, the bucket index mappers of each bucket layer, and the hierarchical bucket index tree includes: In the case where a sorting key in the multidimensional ordered data to be deleted corresponds to a bucket index mapper based on quantization, the value of the sorting key is quantized, and the bucket index corresponding to the sorting key is determined by using the quantized value and the bucket index mapper based on quantization.

10. An apparatus for processing multidimensional ordered data using hierarchical bucketing, characterized in that, The device includes: The parsing module is used to parse the hierarchical bucket configuration information in the hierarchical bucket configuration file and obtain the bucketing strategy information of each bucket layer according to the parsing results; wherein, the number of hierarchical bucket layers is determined by the number of sorting keys in the multidimensional ordered data; A mapper construction module is used to construct a bucket index mapper for each bucket layer according to the bucketing strategy information of each bucket layer; wherein, the bucket index mapper constructed for any bucket layer corresponds to a sorting key in the multidimensional ordered data; The index tree construction module is used to set the number of bucket nodes in each bucket layer in the hierarchical bucket index tree according to the bucketing strategy information of each bucket layer, and to allocate bucket indexes to each bucket node in each bucket layer to generate a hierarchical bucket index tree. The container module is configured to set up storage containers for each leaf bucket node in the hierarchical bucket index tree. The data insertion module is used to determine the leaf bucket node to which the multidimensional ordered data to be added belongs, based on the values ​​of each sort key in the multidimensional ordered data to be added, the bucket index mapper of each bucket layer, and the hierarchical bucket index tree, and to store the multidimensional ordered data to be added in the storage container of its leaf bucket node. The data deletion module is used to determine the leaf bucket node to which the multidimensional ordered data to be deleted belongs based on the values ​​of each sort key in the multidimensional ordered data to be deleted and the bucket index mapper of each bucket layer in the hierarchical bucket index tree, and delete the multidimensional ordered data to be deleted from the storage container of the leaf bucket node to which it belongs.

11. A computer-readable storage medium storing a computer program for performing the method of any one of claims 1-9.

12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-9.

Images