A data processing method and apparatus

Abstract

The application discloses a data processing method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: receiving and analyzing a service request to obtain dimension values corresponding to each dimension, statistical parameter values corresponding to each statistical parameter, and an event time; and inserting a multi-way balanced lookup tree according to the dimension values corresponding to each dimension, the statistical parameter values corresponding to each statistical parameter, and the event time as key-value pairs. The multi-way balanced lookup tree is constructed according to the number of times each dimension is referenced in each feature object, and the feature object defines each dimension and each statistical parameter. The implementation can solve the technical problems of high memory occupancy or untraceable data.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0002] Typically, most of the features required by risk control models reflect the degree of clustering of a certain type of event across certain dimensions. Common clustering features are mainly divided into two categories: deduplication statistical features (counting the number of times different parameters appear under certain same dimensions within a certain period of time) and incremental statistical features (counting the number of times a certain type of event repeats under the same dimension within a certain period of time).

[0003] Currently, there are two common methods for calculating clustered features: map-based algorithms using the original text and hash-based algorithms using two-dimensional arrays. However, both methods have the following technical problems:

[0004] 1) Map-based original text algorithm: High data storage duplication rate, high memory usage, and serious resource consumption;

[0005] 2) Hash-based two-dimensional array algorithm: The relevant data of the feature object cannot be traced back, and there may be a small probability of hash collisions, resulting in inaccurate feature calculation results. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a data processing method and apparatus to solve the technical problems of high memory usage or untraceable data.

[0007] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising:

[0008] Receive and parse business requests to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time;

[0009] Based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, insert a multi-way balanced search tree;

[0010] The multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, and each feature object defines each dimension and each statistical parameter.

[0011] Optionally, before receiving and parsing the business request, the following steps are also included:

[0012] The order of each dimension is determined based on the number of times each dimension is referenced in each feature object;

[0013] Based on the order of the dimensions, a multi-way balanced search tree is constructed, such that heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of the dimensions.

[0014] In the multi-way balanced search tree, "same origin" means that the parent node and child node have the same dimension, while "different origin" means that the parent node and child node have different dimensions.

[0015] Optionally, the order of each dimension is determined based on the number of times each dimension is referenced in each feature object, including:

[0016] A greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object from largest to smallest, thereby determining the order of the dimensions.

[0017] Optionally, a greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object from largest to smallest, including:

[0018] Step 41) Select the largest dimension from the list of feature objects that has not yet been optimized and has the highest number of references;

[0019] Step 42) Based on the maximum dimension, divide the feature object list into a feature object sublist that contains the maximum dimension and a feature object sublist that does not contain the maximum dimension;

[0020] Step 43) For the feature object sublist containing the maximum dimension, move the dimensions that are located before the maximum dimension and appear less frequently than the maximum dimension to the end of the maximum dimension, and mark the maximum dimension as optimized;

[0021] Step 44) Repeat steps 41) to 43) until the largest dimension that has not been optimized and has the most references cannot be selected from the list of feature objects.

[0022] Optionally, a multi-way balanced search tree is inserted based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, including:

[0023] Based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, the target node is determined, and then the statistical parameter values ​​corresponding to each statistical parameter and the event time are used as key-value pairs and inserted into the target node.

[0024] Optionally, based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, a target node is determined, and the statistical parameter values ​​corresponding to each statistical parameter and the event time are used as key-value pairs and inserted into the target node, including:

[0025] Determine the root node of the current dimension in a multi-way balanced search tree;

[0026] In the multi-way balanced search tree, find the same source node of the root node, and use the dimension value of the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and insert it into the same source node;

[0027] In the multi-way balanced search tree, find the heterogeneous node of the root node, and use the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the heterogeneous node.

[0028] Optionally, in the multi-way balanced search tree, a source node of the root node is searched, and the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time are used as key-value pairs to be inserted into the source node, including:

[0029] Step 71) Determine whether the list of same-origin child nodes of the current node is empty; if yes, use the dimension value corresponding to the dimension of the same-origin node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the current node; if no, proceed to step 72).

[0030] Step 72): Take the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and determine whether the key-value pair to be inserted exists in the key-value pair list of the current node; if yes, return the current node directly; if no, proceed to step 73).

[0031] Step 73) Based on the size relationship between the key-value pair to be inserted and the existing key-value pairs in the current node, continue to search for the node to be inserted from the same source left child node or same source right child node of the existing key-value pair in the current node, and then insert the key-value pair to be inserted into it.

[0032] Optionally, in the multi-way balanced search tree, the heterogeneous node of the root node is searched, and the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time are used as key-value pairs to be inserted into the heterogeneous node, including:

[0033] Determine whether there exists a heterogeneous child node in the list of heterogeneous child nodes corresponding to the key-value pair to be inserted, with the same dimension as the dimension to which the key-value pair to be inserted belongs;

[0034] If so, then the heterogeneous child node will be used as the new root node;

[0035] If not, initialize a new heterogeneous child node, insert the new heterogeneous child node into the heterogeneous child node list of key-value pairs corresponding to the root node, so that each heterogeneous child node in the heterogeneous child node list is arranged in ascending order according to its corresponding dimension; and, use the new heterogeneous child node as the new root node.

[0036] In the multi-way balanced search tree, find the same source node of the new root node, and use the dimension value corresponding to the dimension of the same source node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the same source node.

[0037] Optionally, after inserting the multi-way balanced search tree using the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, the method further includes:

[0038] The number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source.

[0039] Optionally, the number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source, including:

[0040] Locate the logical root node of the node containing the key-value pair corresponding to the statistical parameter value and the event time in the multi-way balanced search tree;

[0041] The key-value pairs in the logical root node and its child nodes are traversed to calculate the number of business requests corresponding to each feature object, and the business requests are traced by dimension value.

[0042] Optionally, the number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source, including:

[0043] The number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced back to their dimensional values.

[0044] Optionally, the number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced by dimension value, including:

[0045] Step 121) Determine whether the current node is the node corresponding to the statistical parameter value and the event time; if yes, proceed to step 122); if no, proceed to step 123.

[0046] Step 122) Traverse the key-value pairs in the current node and its child nodes to calculate the number of business requests corresponding to each feature object, and trace the business requests by dimension value.

[0047] Step 123) Recursively select the heterogeneous child nodes of the current node as the new current node and execute step 121).

[0048] Additionally, according to another aspect of the present invention, a data processing apparatus is provided, comprising:

[0049] The parsing module is used to receive and parse business requests to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time.

[0050] The processing module is used to insert a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs;

[0051] The multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, and each feature object defines each dimension and each statistical parameter.

[0052] Optionally, it also includes a building module for:

[0053] The order of each dimension is determined based on the number of times each dimension is referenced in each feature object;

[0054] Based on the order of the dimensions, a multi-way balanced search tree is constructed, such that heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of the dimensions.

[0055] In the multi-way balanced search tree, "same origin" means that the parent node and child node have the same dimension, while "different origin" means that the parent node and child node have different dimensions.

[0056] Optionally, the building module is further configured to:

[0057] A greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object from largest to smallest, thereby determining the order of the dimensions.

[0058] Optionally, the building module is further configured to:

[0059] Step 41) Select the largest dimension from the list of feature objects that has not yet been optimized and has the highest number of references;

[0060] Step 42) Based on the maximum dimension, divide the feature object list into a feature object sublist that contains the maximum dimension and a feature object sublist that does not contain the maximum dimension;

[0061] Step 43) For the feature object sublist containing the maximum dimension, move the dimensions that are located before the maximum dimension and appear less frequently than the maximum dimension to the end of the maximum dimension, and mark the maximum dimension as optimized;

[0062] Step 44) Repeat steps 41) to 43) until the largest dimension that has not been optimized and has the most references cannot be selected from the list of feature objects.

[0063] Optionally, the processing module is further configured to:

[0064] Based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, the target node is determined, and then the statistical parameter values ​​corresponding to each statistical parameter and the event time are used as key-value pairs and inserted into the target node.

[0065] Optionally, the processing module is further configured to:

[0066] Determine the root node of the current dimension in a multi-way balanced search tree;

[0067] In the multi-way balanced search tree, find the same source node of the root node, and use the dimension value of the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and insert it into the same source node;

[0068] In the multi-way balanced search tree, find the heterogeneous node of the root node, and use the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the heterogeneous node.

[0069] Optionally, the processing module is further configured to:

[0070] Step 71) Determine whether the list of same-origin child nodes of the current node is empty; if yes, use the dimension value corresponding to the dimension of the same-origin node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the current node; if no, proceed to step 72).

[0071] Step 72): Take the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and determine whether the key-value pair to be inserted exists in the key-value pair list of the current node; if yes, return the current node directly; if no, proceed to step 73).

[0072] Step 73) Based on the size relationship between the key-value pair to be inserted and the existing key-value pairs in the current node, continue to search for the node to be inserted from the same source left child node or same source right child node of the existing key-value pair in the current node, and then insert the key-value pair to be inserted into it.

[0073] Optionally, the processing module is further configured to:

[0074] Determine whether there exists a heterogeneous child node in the list of heterogeneous child nodes corresponding to the key-value pair to be inserted, with the same dimension as the dimension to which the key-value pair to be inserted belongs;

[0075] If so, then the heterogeneous child node will be used as the new root node;

[0076] If not, initialize a new heterogeneous child node, insert the new heterogeneous child node into the heterogeneous child node list of key-value pairs corresponding to the root node, so that each heterogeneous child node in the heterogeneous child node list is arranged in ascending order according to its corresponding dimension; and, use the new heterogeneous child node as the new root node.

[0077] In the multi-way balanced search tree, find the same source node of the new root node, and use the dimension value corresponding to the dimension of the same source node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the same source node.

[0078] Optionally, it also includes a computing module for:

[0079] Using the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, after inserting them into a multi-way balanced search tree, the number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source.

[0080] Optionally, the computing module is further configured to:

[0081] Locate the logical root node of the node containing the key-value pair corresponding to the statistical parameter value and the event time in the multi-way balanced search tree;

[0082] The key-value pairs in the logical root node and its child nodes are traversed to calculate the number of business requests corresponding to each feature object, and the business requests are traced by dimension value.

[0083] Optionally, the computing module is further configured to:

[0084] The number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced back to their dimensional values.

[0085] Optionally, the computing module is further configured to:

[0086] Step 121) Determine whether the current node is the node corresponding to the statistical parameter value and the event time; if yes, proceed to step 122); if no, proceed to step 123.

[0087] Step 122) Traverse the key-value pairs in the current node and its child nodes to calculate the number of business requests corresponding to each feature object, and trace the business requests by dimension value.

[0088] Step 123) Recursively select the heterogeneous child nodes of the current node as the new current node and execute step 121).

[0089] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0090] One or more processors;

[0091] Storage device for storing one or more programs.

[0092] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0093] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0094] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0095] One embodiment of the above invention has the following advantages or beneficial effects: Because it employs a technique of inserting a multi-way balanced search tree based on the dimension values, statistical parameter values, and event times of each dimension as key-value pairs, and where the multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, it overcomes the technical problems of high memory consumption or lack of data traceability in existing technologies. This embodiment of the invention creates a multi-way balanced search tree based on the number of references of each dimension and inserts dimension values, statistical parameter values, and event times as key-value pairs, supporting data reuse from the underlying layer to save memory resources. Furthermore, the relevant data of the feature objects is traceable, and the calculation results of the feature objects are accurate.

[0096] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0097] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0098] Figure 1 This is a schematic diagram of a general risk control system technical solution;

[0099] Figure 2 This is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention;

[0100] Figure 3 This is a schematic diagram of the structure of a multi-way balanced search tree according to an embodiment of the present invention;

[0101] Figure 4 This is a schematic diagram of the main process of optimizing each dimension using a greedy optimization algorithm according to an embodiment of the present invention;

[0102] Figure 5 This is a schematic diagram of the main flow of a data processing method according to a possible embodiment of the present invention;

[0103] Figure 6 This is a schematic diagram of the main flow of the feature insertion algorithm according to an embodiment of the present invention;

[0104] Figure 7 This is a schematic diagram of the main flow of the same-origin node query algorithm according to an embodiment of the present invention;

[0105] Figure 8This is a schematic diagram of the main flow of the heterogeneous node query algorithm according to an embodiment of the present invention;

[0106] Figure 9 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention;

[0107] Figure 10 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0108] Figure 11 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0109] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0110] Risk control is the strongest guarantee for the effective implementation of every enterprise's strategic business and the accurate reach of target users. Behind each business, there are usually several risk control models, and each risk control model must have enough features to ensure that the model does not have "false positives" or "false negatives". Therefore, how to efficiently process the features required by the model is a problem of great concern.

[0111] As a pure streaming data processing engine, Flink is frequently introduced into feature processing scenarios for risk control models due to its advantages such as real-time performance, strong consistency, and high scalability. Typical technical solutions for risk control systems include... Figure 1 As shown, the Flink feature system is responsible for real-time feature processing, the model center completes the identification of risk events and outputs the identification results to the downstream, the decision system makes risk judgments based on the risk identification results and business decision criteria (which can usually be changed in real time), and finally the business processing system performs relevant business risk handling (such as order deletion, blacklisting, etc.).

[0112] Generally speaking, most of the features required by risk control models reflect the degree of clustering of a certain type of event in certain dimensions. Common clustering features are mainly divided into two categories: deduplication statistical features (counting the number of times different parameters appear under certain same dimensions within a certain period of time) and incremental statistical features (counting the number of times a certain type of event repeats under the same dimension within a certain period of time).

[0113] Currently, there are two common methods for calculating clustered features: the original text algorithm based on map and the two-dimensional array algorithm based on hash. The following example uses the deduplication feature "counting the number of different accounts with the same order IP, product SKU, and delivery phone number location within 12 hours" to illustrate the specific implementation logic of the two algorithms. Example data is shown in Table 1.

[0114] Table 1 Example Data

[0115]

[0116]

[0117] Map-based plaintext algorithms refer to storing all feature-related data (including dimension values, parameter values, and event times) directly in plaintext form into a map structure (dimensional value combinations serve as keys, and event times and statistical parameter values ​​are presented as lists as values) to perform calculations. The data storage format in map-based plaintext algorithms, as shown in Table 1, is illustrated in Table 2.

[0118] Table 2 Storage format of the original text algorithm based on map

[0119]

[0120] The hash-based two-dimensional array algorithm refers to converting all feature-related data (including dimension values ​​and parameter values) into hashes and storing the resulting hash values ​​in a two-dimensional array to perform calculations. The storage format of the data shown in Table 1 in the hash-based two-dimensional array algorithm is shown in Table 3. In the two-dimensional array, the element ARR[x][0] (where x is the index of the first dimension of the two-dimensional array, x>=0) always stores the hash value of the dimension value combination (usually, all dimension values ​​are concatenated in a certain order). The elements ARR[x][2k-1] and ARR[x][2k] (where 2k-1 and 2k correspond to the second dimension indices of the two-dimensional array, k>=1) store the hash values ​​of the corresponding statistical parameter values ​​and the event timestamps, respectively. That is, in the hash-based two-dimensional array algorithm, feature data with the same dimension values ​​have the same first dimension index in the two-dimensional array.

[0121] Table 3 Storage format of hash-based two-dimensional array algorithm

[0122]

[0123] Tables 2 and 3 show the feature calculation methods of the original text algorithm based on map and the two-dimensional array algorithm based on hash, respectively. The advantages and disadvantages of the two algorithms can be clearly seen from them, as summarized in Table 4.

[0124] Table 4. Advantages and disadvantages of the two algorithms

[0125]

[0126] Since Flink computes feature objects based on the machine's local memory at runtime, map-based algorithms are typically only suitable for feature processing of models that require data backtracking, have high accuracy requirements for feature computation results, and have very small business volume. On the other hand, hash-based two-dimensional array algorithms are suitable for the opposite scenarios, only for feature processing of models with large business volume, no need for data backtracking, and low accuracy requirements for feature computation results.

[0127] Therefore, it is evident that the two existing algorithms are only applicable to two extreme scenarios. However, most scenarios encountered in production environments for risk control involve large-scale business operations, high precision requirements for feature calculation results, and frequent data backtracking. For example, a common risk control model logic in the login scenarios of large internet companies is: "If more than 20 different accounts with the same device ID and IP address log in within 6 hours, these requests are considered risk events, requiring a 'kick-out' operation for all accounts, and simultaneously blacklisting the corresponding device ID, IP address, and all requested accounts." This risk control model logic is characterized by its large scale, high precision requirements for feature calculation results, and the need to support feature data backtracking. Clearly, neither of the two existing feature calculation methods can adequately meet these requirements.

[0128] To address the technical problems existing in the prior art, the embodiments of the present invention start from the underlying data structure and use a multi-way balanced search tree to store dimension values, statistical parameter values ​​and event times. This supports data reuse from the bottom layer to save memory resources and also enables data traceability.

[0129] Figure 2 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention. As one embodiment of the present invention, such as... Figure 2 As shown, the data processing method may include:

[0130] Step 201: Receive and parse the business request to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time.

[0131] First, the system receives business requests from the upstream system and parses them to obtain the dimension values, statistical parameter values, and event times corresponding to each dimension.

[0132] Taking the data shown in Table 1 as an example, the order IP, product SKU, and mobile phone location are the dimensions, and the account is the statistical parameter. Taking the first row of data as an example, the corresponding dimension values ​​are 11.10.34.18, 100010345793, and Beijing. The corresponding statistical parameter value is wx_ajdevaevds, and the event time is 2021-09-02 08:10:04. Taking the second row of data as an example, the corresponding dimension values ​​are 11.10.34.18, 100010345793, and Beijing. The corresponding statistical parameter value is yhd_jhnads987, and the event time is 2021-09-02 09:10:13.

[0133] It should be noted that the dimensions and statistical parameters to be parsed from the business request are determined by the feature object. In other words, the feature object defines each dimension and each statistical parameter. Taking the deduplication statistical feature "counting the number of different accounts with the same ordering IP, product SKU, and delivery phone number location within 12 hours" as an example, the dimensions defined by this feature object are ordering IP, product SKU, and phone number location, and the statistical parameter is account.

[0134] Step 202: Insert a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs.

[0135] After parsing the dimension values, statistical parameter values, and event times corresponding to each dimension from the business request, these dimension values, statistical parameter values, and event times are used as key-value pairs and inserted into a multi-way balanced search tree (i.e., a B-tree).

[0136] To support data reuse from the underlying layer and thus save memory resources, embodiments of the present invention create a multi-way balanced search tree based on the number of references for each dimension. Therefore, in embodiments of the present invention, the multi-way balanced search tree is constructed based on the number of references for each dimension in each feature object, where each feature object defines each dimension and each statistical parameter.

[0137] Optionally, before step 201, the process of constructing a multi-way balanced search tree is further included. Specifically, this includes: determining the order of each dimension based on the number of times each dimension is referenced in each feature object; constructing a multi-way balanced search tree based on the order of the dimensions, such that heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of the dimensions; wherein, in the multi-way balanced search tree, homogeneous means that the parent node and child node have the same dimension, and heterogeneous means that the parent node and child node have different dimensions.

[0138] The purpose of this invention is to achieve data reuse through a multi-way balanced search tree. To maximize memory saving, some optimizations are needed in the dimensional definition of the feature objects. Table 5 shows two dimensional definition methods for four feature objects. Clearly, "Dimension Definition 2" performs better in terms of memory saving than "Dimension Definition 1".

[0139] Table 5 Comparison of Dimensional Definitions of Feature Objects

[0140] Feature name Dimension Definition 1 Dimension Definition 2 Feature object 1 C, A, B, D A, B, D, C Feature object 2 D, B, A, E A, B, D, E Feature object 3 H, A, B A, B, H Feature object 4 A, C, F A, C, F

[0141] Therefore, in this embodiment of the invention, the order of reference counts of each dimension from largest to smallest is determined based on the number of times each dimension is referenced in each feature object, thereby constructing a multi-way balanced search tree based on the order of reference counts of each dimension from largest to smallest, thereby achieving the goal of maximizing memory saving.

[0142] However, because each feature object is defined independently, and model developers often define feature objects arbitrarily, it's difficult to adhere to the strict requirements of "Dimension Definition 2." To maximize memory efficiency, some optimization of the dimensions of the feature objects defined by model developers is necessary. Optionally, the order of each dimension can be determined based on its reference count in each feature object. This includes using a greedy optimization algorithm to sort the dimensions based on their reference count in descending order, thereby determining the order of the dimensions.

[0143] This invention starts from the global perspective of the feature object, sorts the dimensions in each feature object in descending order according to the number of times they are referenced in the feature object, and then constructs a multi-way balanced search tree based on the sorting of each dimension, so that the heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of each dimension.

[0144] like Figure 3 As shown, the multi-way balanced search tree constructed in this embodiment of the invention has the following characteristics of a traditional B-tree:

[0145] 1) Each node can store multiple key-value pairs, which are arranged in ascending order; and for each key-value pair, all key-value pairs in its left child node must be smaller than it, and all key-value pairs in its right child node must be larger than it.

[0146] 2) Except for the root node, the number of key-value pairs in all nodes satisfies the formula: Where m is the degree of the tree, and k is the number of key-value pairs in the node;

[0147] 3) Except for leaf nodes, the number of key-value pairs and the number of child nodes in all nodes satisfy the formula: n = k + 1, where n is the number of child nodes of a node and k is the number of key-value pairs in a node.

[0148] However, unlike traditional B-trees, the multi-way balanced search tree constructed in this embodiment of the invention introduces the concept of dimension. Therefore, the multi-way balanced search tree constructed in this embodiment of the invention (which can be called a multi-dimensional B-tree for ease of description) also has the following characteristics:

[0149] 1) In a multi-dimensional B-tree, each node belongs to a certain dimension, such as IP address, mobile phone number, and mobile phone location; at the same time, the key-value pairs stored in the node also belong to a certain dimension and are consistent with the dimension of the corresponding node.

[0150] 2) In a multi-dimensional B-tree, there are usually multiple dimensions, and the relationships between these dimensions typically correspond to the dimensional relationships defined for one or more feature objects; among them, Figure 3 This demonstrates how dimension values ​​are stored for a given dimension, A and B (note the order of the dimensions).

[0151] 3) In a multi-dimensional B-tree, the relationship between parent and child nodes is divided into same-origin and different-origin. Same-origin means that the parent and child nodes have the same dimension, while different-origin means that the parent and child nodes have different dimensions. A different-origin child node is usually attached to a key-value pair of its different-origin parent node; for example, in... Figure 3 In the middle, the node containing key-value pairs B11 and B19 is attached to the A10 key-value pair of its heterogeneous parent node;

[0152] 4) In a multi-dimensional B-tree, there is only one physical root node, and the dimension of the physical root node must be the first dimension in the definition of one or more feature objects (i.e., the dimension with the most references); for example, Figure 3 The node containing the key-value pair A10, A40, and A70 is the physical root node;

[0153] 5) In a multi-dimensional B-tree, there are usually multiple logical root nodes, and the dimension of the logical root node cannot be the first dimension in the feature object definition; furthermore, the tree structures corresponding to multiple logical root nodes with the same dimension are independent of each other (they do not interfere with each other when adjusting the tree structure); for example, in Figure 3 In the tree, the two nodes containing key-value pairs B11, B19 and B21, B23, B25 are both logical root nodes, and the tree structures corresponding to these two logical root nodes are independent of each other.

[0154] 6) During the insertion and deletion process of a multi-dimensional B-tree, only same parent-child nodes and same sibling nodes will participate in the tree structure adjustment to ensure that the tree meets the definition of a B-tree, but different parent-child nodes and different sibling nodes will not participate in the tree structure adjustment.

[0155] Optionally, such as Figure 4 As shown, a greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object from largest to smallest, including:

[0156] Step 401: Determine whether the largest dimension that has not yet been optimized and has the most references can be selected from the feature object list; if yes, proceed to step 402; otherwise, end.

[0157] Specifically, select the dimension that has not yet been optimized and has the highest number of references from the list of feature objects; this is the maximum dimension. If no maximum dimension is selected, it indicates that the entire optimization process has ended.

[0158] Step 402: Based on the maximum dimension, divide the feature object list into a feature object sublist that includes the maximum dimension and a feature object sublist that does not include the maximum dimension.

[0159] In this step, the feature object list in step 401 is divided into two feature object sublists based on the selected maximum dimension: a feature object sublist whose feature object definition contains the maximum dimension and a feature object sublist whose feature object definition does not contain the maximum dimension.

[0160] Step 403: For the feature object sublist containing the maximum dimension, move the dimensions that are located before the maximum dimension and appear less frequently than the maximum dimension to the end of the maximum dimension, and mark the maximum dimension as optimized.

[0161] For each feature object sublist containing a maximum dimension in its feature object definition, optimize the feature objects in the list. Specifically, move the dimensions that precede the maximum dimension in the feature object definition and have fewer references than the maximum dimension to the end of the maximum dimension. During the optimization process, mark the maximum dimension of each related feature object as "optimized".

[0162] Next, for the two feature object sublists in step 402, the greedy optimization algorithm is called again for optimization, that is, steps 401 to 403 are repeated until the largest dimension that has not been optimized and has the most references cannot be selected from the feature object list.

[0163] Based on the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problems of high memory consumption or data untraceability in the prior art by inserting a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs. The multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object. The embodiments of the present invention create a multi-way balanced search tree based on the number of references to each dimension and insert dimension values, statistical parameter values, and event time as key-value pairs, supporting data reuse from the underlying level to save memory resources. Furthermore, the relevant data of the feature object is traceable, and the calculation results of the feature object are accurate.

[0164] Figure 5 This is a schematic diagram of the main flow of a data processing method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 5 As shown, the data processing method may include:

[0165] Step 501: Using a greedy optimization algorithm and based on the order of the number of times each dimension is referenced in each feature object from largest to smallest, sort the dimensions to determine the order of the dimensions.

[0166] Step 502: Construct a multi-way balanced search tree according to the order of the dimensions, so that the heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of the dimensions.

[0167] Step 503: Receive and parse the business request to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time.

[0168] Step 504: Insert a multi-way balanced search tree using the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs.

[0169] After parsing the dimension values, statistical parameter values, and event times corresponding to each dimension from the business request, these dimension values, statistical parameter values, and event times are used as key-value pairs and inserted into a multi-way balanced search tree.

[0170] Optionally, step 504 may include: determining the target node based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, and then inserting the statistical parameter values ​​corresponding to each statistical parameter and the event time as key-value pairs into the target node. As mentioned above, the data involved in the execution of the feature object mainly includes three parts: dimension values, statistical parameter values, and event time. Therefore, in this step, a feature insertion algorithm is used to convert each dimension value, each parameter value, and the event time into key-value pairs, and insert them into the multi-way balanced search tree in the optimized dimension order (the key-value pairs corresponding to the statistical parameter values ​​and the event time are at the end). Specifically, the feature insertion algorithm determines the target node based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, and then inserts the statistical parameter values ​​corresponding to each statistical parameter and the event time as key-value pairs into the target node.

[0171] Optionally, determining the target node based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, and then inserting the statistical parameter values ​​corresponding to each statistical parameter and the event time as key-value pairs into the target node, may include: determining the root node of the current dimension in the multi-way balanced search tree; searching for homologous nodes of the root node in the multi-way balanced search tree, and inserting the dimension value of the current dimension or the statistical values ​​corresponding to each statistical parameter and the event time as key-value pairs to be inserted into the homologous node; searching for dissimilar nodes of the root node in the multi-way balanced search tree, and inserting the dimension value corresponding to the dimension of the dissimilar node or the statistical values ​​corresponding to each statistical parameter and the event time as key-value pairs to be inserted into the dissimilar node.

[0172] In this embodiment of the invention, a feature insertion algorithm is used to convert each dimension value, each parameter value, and event time into key-value pairs, and then insert them into a multi-way balanced search tree in the optimized dimension order (the key-value pairs corresponding to the statistical parameter values ​​and event times are located at the end). The feature insertion algorithm consists of three parts: feature object insertion algorithm, same-source node query algorithm, and different-source node query algorithm.

[0173] The feature object insertion algorithm has three input parameters:

[0174] btNode is a data pre-insertion node (the actual data inserted may not be this node), and the value passed to it during its initial call is the root node of the multi-way balanced search tree;

[0175] featureItem is a feature object that includes dimension values, statistical parameter values, and event time.

[0176] insertIdx is a pointer to the currently inserted data, pointing to the values ​​of various dimensions or statistical parameters and the event time. The initial value passed to it is 0.

[0177] like Figure 6 As shown, the feature insertion algorithm includes the following execution steps:

[0178] Step 601: Calculate the corresponding dimension and key-value pair based on featureItem and insertIdx; where the key-value pair stores the values ​​of each dimension or a value composed of statistical parameter values ​​and event time.

[0179] Step 602: Based on the calculated dimensions and key-value pairs, query the source nodes from the pre-insertion nodes to find the actual source nodes for the current key-value pair. The returned value may be the pre-insertion node itself, or its source child nodes, grandchild nodes, etc. The query logic for source nodes can be implemented using a source node query algorithm.

[0180] Step 603: Insert the key-value pair into the same-origin node retrieved in step 602. The insertion operation will return a node. A split operation may occur during the insertion process, so the returned node may not be the node retrieved in step 602. The specific method for inserting key-value pairs is consistent with the insertion logic of a traditional B-tree, and will not be elaborated further in this embodiment.

[0181] Step 604: Determine whether insertIdx has exceeded the length of the dimension value key-value pair and the statistical parameter and event time key-value pair; if yes, it means that all data of the current feature object has been inserted, return to the node returned during insertion in step 603, and the process ends; otherwise, continue to execute step 605.

[0182] Step 605: Calculate the dimension to which the next key-value pair to be inserted belongs, and at the same time find the pre-insertion node (logical root node) of the next key-value pair from the inserted key-value pairs; wherein, the query logic of heterogeneous nodes can be implemented by heterogeneous node query algorithm.

[0183] Next, the feature object insertion algorithm is called again for the next key-value pair to insert the feature object and update the heterogeneous parent node of the returned node.

[0184] To ensure that the entire tree satisfies the definition of a multi-dimensional B-tree, the key-value pair is not directly inserted into the specified node during the insertion operation, but rather needs to be inserted into a reasonable node; therefore, the same-origin node query algorithm can be used to retrieve reasonable nodes.

[0185] Optionally, such as Figure 7As shown, in the multi-way balanced search tree, the same-source node of the root node is searched, and the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time are used as the key-value pair to be inserted and inserted into the same-source node, including:

[0186] Step 701: Determine if the list of sibling nodes of the current node is empty; if yes, proceed to step 702; if no, proceed to step 703.

[0187] Step 702: Use the dimension value corresponding to the dimension of the same source node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, insert it into the current node and return.

[0188] Step 703: Take the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and determine whether the key-value pair to be inserted exists in the key-value pair list of the current node; if yes, then proceed to step 704; if no, then proceed to step 705.

[0189] Step 704: Return directly to the current node.

[0190] Step 705: Based on the size relationship between the key-value pair to be inserted and the existing key-value pairs in the current node, continue to search for the node to be inserted from the same source left child node or same source right child node of the existing key-value pair in the current node, and then insert the key-value pair to be inserted into it and return.

[0191] Specifically, the same-origin node query algorithm has three input parameters: btNode, the starting node to be queried; source, the dimension to which the key-value pair to be inserted belongs; and valuePair, the key-value pair to be inserted.

[0192] Optionally, the same-origin node query algorithm includes the following execution steps:

[0193] 1) Check if the list of sibling nodes of the current node is empty; if it is empty, it means that the current node is a leaf node, so just return to the current node and insert (there may be a split operation when performing the insertion operation, that is, the final node of the key-value pair insertion may not be the current node); otherwise, proceed to the next step;

[0194] 2) Try to find the key-value pair to be inserted in the key-value pair list of the current node; if the key-value pair to be inserted is contained in the key-value pair list of the current node, simply return to the current node; otherwise, proceed to the next step.

[0195] 3) Based on the characteristics of a traditional B-tree, and considering the relationship between the key-value pair to be inserted and the existing key-value pairs in the current node, continue searching for the node to be inserted from the same-origin left or right child node of a key-value pair in the current node (located by the midIdx pointer) and return it.

[0196] In the multi-dimensional B-tree of this invention, the association between key-value pairs of different dimensions is established by associating a key-value pair in a heterogeneous parent node with a heterogeneous child node. Furthermore, the nodes in the heterogeneous child node list are arranged in ascending order according to their respective dimensions. Therefore, during node insertion, it is necessary to find the node to which the heterogeneous key-value pair should be inserted. The heterogeneous node query algorithm provided in this invention can find the node to which the heterogeneous key-value pair should be inserted. It should be noted that the node returned by the heterogeneous node query algorithm is only a logical root node; the node where the data is ultimately inserted may not be this node.

[0197] Optionally, such as Figure 8 As shown, in the multi-way balanced search tree, the heterogeneous node of the root node is searched, and the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time are used as the key-value pair to be inserted into the heterogeneous node, including:

[0198] Step 801: Determine whether there is a heterogeneous child node in the list of heterogeneous child nodes corresponding to the key-value pair of the root node with the same dimension as the key-value pair to be inserted; if yes, proceed to step 802; if no, proceed to step 803.

[0199] Step 802: The heterogeneous child node is used as the new root node.

[0200] Step 803: Initialize new heterogeneous child nodes by inserting the new heterogeneous child nodes into the heterogeneous child node list of key-value pairs corresponding to the root node, so that each heterogeneous child node in the heterogeneous child node list is arranged in ascending order according to its corresponding dimension; and, use the new heterogeneous child node as the new root node.

[0201] Step 804: In the multi-way balanced search tree, find the same source node of the new root node, and use the dimension value corresponding to the dimension of the same source node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the same source node.

[0202] The heterogeneous node query algorithm has two parameters: parentValuePair, the key-value pair corresponding to the previous dimension; and heteroSource, the dimension to which the current key-value pair to be inserted belongs.

[0203] Optionally, the heterogeneous node query algorithm includes the following execution steps:

[0204] 1) Use a binary search algorithm to search for a heterogeneous child node with dimension heterSource in the list of heterogeneous child nodes of a specified key-value pair. If found, return the corresponding node directly; otherwise, proceed to the next step.

[0205] 2) Initialize new heterogeneous child nodes;

[0206] 3) Insert the heterogeneous node into the appropriate position in the heterogeneous child node list of the specified key-value pair to maintain the heterogeneous child node list in ascending order of source;

[0207] 4) Return the new heterogeneous child node.

[0208] Furthermore, the specific implementation details of the data processing method in one of the reference embodiments of the present invention have been described in detail in the data processing method described above, so the details will not be repeated here.

[0209] In an embodiment of the present invention, after inserting a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, the method further includes: calculating the number of business requests corresponding to each feature object based on the multi-way balanced search tree, and tracing the data source of the business requests.

[0210] In the embodiments of the present invention, the triggering methods for feature object calculation are generally divided into two types: event triggering and time triggering.

[0211] Optionally, the number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source. This includes: locating the logical root node of the node containing the key-value pair corresponding to the statistical parameter value and the event time in the multi-way balanced search tree; traversing the key-value pairs in the logical root node and its child nodes to calculate the number of business requests corresponding to each feature object, and tracing the business requests back to their source by dimension value. In event-triggered feature object calculations, all event-related information already exists, such as the various dimension values, various statistical parameter values ​​(excluding other associated statistical parameters), event time, and the nodes where the various dimension values ​​are stored. Therefore, the implementation of event-triggered feature object calculations is relatively simple; it only requires traversing the logical root node and its child nodes of the node containing the key-value pair corresponding to the statistical parameter value and the event time to complete the feature object calculation and the data backtracking of the statistical parameter values.

[0212] Optionally, the number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source. This includes: calculating the number of business requests corresponding to each feature object by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and tracing the business requests back to their source by dimension values. The time-triggered method refers to the calculation of feature objects being periodically triggered by time. At the time of triggering, there is no other data information besides the feature object definition and the trigger time. Therefore, the time-triggered calculation method is relatively complex to implement, requiring the backtracking of all relevant data while calculating the result. Furthermore, the result returned by this calculation method is typically a list containing different dimension values.

[0213] Optionally, the number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced by dimension value, including:

[0214] Step 121) Determine whether the current node is the node corresponding to the statistical parameter value and the event time; if yes, proceed to step 122); if no, proceed to step 123.

[0215] Step 122) Traverse the key-value pairs in the current node and its child nodes to calculate the number of business requests corresponding to each feature object, and trace the business requests by dimension value.

[0216] Step 123) Recursively select the heterogeneous child nodes of the current node as the new current node and execute step 121).

[0217] The time-triggered feature object calculation method is based on depth-first traversal. The algorithm has three input parameters: btNode, the node; feature, the feature object definition, which contains information about each dimension of the feature object; computeIdx, the dimension index to be calculated, which is passed in as 0 on the initial call.

[0218] The feature object calculation method mainly includes the following execution steps:

[0219] 1) Initialize the feature result list;

[0220] 2) Determine if the dimension of the current node is consistent with the dimension specified by computeIdx in the feature object definition; if yes, continue to step 3); if no, return an empty result list.

[0221] 3) Determine whether the current node has reached the node corresponding to the statistical parameter value and event time based on computeIdx; if so, complete the calculation based on the current node and its child nodes (the specific calculation method is related to the statistical method of the feature object), and store the calculation result in the feature result list; otherwise, continue to step 4);

[0222] 4) Recursively call the feature object calculation method on all heterogeneous child nodes of the current node, and complete the source tracing operation of the current dimension value in the process;

[0223] 5) Recursively call the feature object calculation method on all homologous child nodes of the current node;

[0224] 6) Return the feature results list.

[0225] Therefore, this invention creates a multi-way balanced search tree based on the number of references for each dimension and inserts dimension values, statistical parameter values, and event times as key-value pairs. This supports data reuse from the bottom layer to save memory resources. Moreover, the relevant data of the feature object is traceable, and the calculation results of the feature object are accurate. This invention can meet the feature processing needs of most risk control scenario models.

[0226] Figure 9 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention. Figure 9 As shown, the data processing device 900 includes a parsing module 901 and a processing module 902; wherein, the parsing module 901 is used to receive and parse business requests to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time; the processing module 902 is used to insert a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs; wherein, the multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, and the feature object defines each dimension and each statistical parameter.

[0227] Optionally, it also includes a building module for:

[0228] The order of each dimension is determined based on the number of times each dimension is referenced in each feature object;

[0229] Based on the order of the dimensions, a multi-way balanced search tree is constructed, such that heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of the dimensions.

[0230] In the multi-way balanced search tree, "same origin" means that the parent node and child node have the same dimension, while "different origin" means that the parent node and child node have different dimensions.

[0231] Optionally, the building module is further configured to:

[0232] A greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object from largest to smallest, thereby determining the order of the dimensions.

[0233] Optionally, the building module is further configured to:

[0234] Step 41) Select the largest dimension from the list of feature objects that has not yet been optimized and has the highest number of references;

[0235] Step 42) Based on the maximum dimension, divide the feature object list into a feature object sublist that contains the maximum dimension and a feature object sublist that does not contain the maximum dimension;

[0236] Step 43) For the feature object sublist containing the maximum dimension, move the dimensions that are located before the maximum dimension and appear less frequently than the maximum dimension to the end of the maximum dimension, and mark the maximum dimension as optimized;

[0237] Step 44) Repeat steps 41) to 43) until the largest dimension that has not been optimized and has the most references cannot be selected from the list of feature objects.

[0238] Optionally, the processing module 902 is further configured to:

[0239] Based on the mounting order of the multi-way balanced search tree and the dimension values ​​corresponding to each dimension, the target node is determined, and then the statistical parameter values ​​corresponding to each statistical parameter and the event time are used as key-value pairs and inserted into the target node.

[0240] Optionally, the processing module 902 is further configured to:

[0241] Determine the root node of the current dimension in a multi-way balanced search tree;

[0242] In the multi-way balanced search tree, find the same source node of the root node, and use the dimension value of the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and insert it into the same source node;

[0243] In the multi-way balanced search tree, find the heterogeneous node of the root node, and use the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the heterogeneous node.

[0244] Optionally, the processing module 902 is further configured to:

[0245] Step 71) Determine whether the list of same-origin child nodes of the current node is empty; if yes, use the dimension value corresponding to the dimension of the same-origin node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the current node; if no, proceed to step 72).

[0246] Step 72): Take the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and determine whether the key-value pair to be inserted exists in the key-value pair list of the current node; if yes, return the current node directly; if no, proceed to step 73).

[0247] Step 73) Based on the size relationship between the key-value pair to be inserted and the existing key-value pairs in the current node, continue to search for the node to be inserted from the same source left child node or same source right child node of the existing key-value pair in the current node, and then insert the key-value pair to be inserted into it.

[0248] Optionally, the processing module 902 is further configured to:

[0249] Determine whether there exists a heterogeneous child node in the list of heterogeneous child nodes corresponding to the key-value pair to be inserted, with the same dimension as the dimension to which the key-value pair to be inserted belongs;

[0250] If so, then the heterogeneous child node will be used as the new root node;

[0251] If not, initialize a new heterogeneous child node, insert the new heterogeneous child node into the heterogeneous child node list of key-value pairs corresponding to the root node, so that each heterogeneous child node in the heterogeneous child node list is arranged in ascending order according to its corresponding dimension; and, use the new heterogeneous child node as the new root node.

[0252] In the multi-way balanced search tree, find the same source node of the new root node, and use the dimension value corresponding to the dimension of the same source node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the same source node.

[0253] Optionally, it also includes a computing module for:

[0254] Using the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, after inserting them into a multi-way balanced search tree, the number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source.

[0255] Optionally, the computing module is further configured to:

[0256] Locate the logical root node of the node containing the key-value pair corresponding to the statistical parameter value and the event time in the multi-way balanced search tree;

[0257] The key-value pairs in the logical root node and its child nodes are traversed to calculate the number of business requests corresponding to each feature object, and the business requests are traced by dimension value.

[0258] Optionally, the computing module is further configured to:

[0259] The number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced back to their dimensional values.

[0260] Optionally, the computing module is further configured to:

[0261] Step 121) Determine whether the current node is the node corresponding to the statistical parameter value and the event time; if yes, proceed to step 122); if no, proceed to step 123.

[0262] Step 122) Traverse the key-value pairs in the current node and its child nodes to calculate the number of business requests corresponding to each feature object, and trace the business requests by dimension value.

[0263] Step 123) Recursively select the heterogeneous child nodes of the current node as the new current node and execute step 121).

[0264] It should be noted that the specific implementation details of the data processing device described in this invention have been described in detail in the data processing method described above, so the details will not be repeated here.

[0265] Figure 10 An exemplary system architecture 1000 is shown for which the data processing method or data processing apparatus of the present invention can be applied.

[0266] like Figure 10 As shown, system architecture 1000 may include terminal devices 1001, 1002, and 1003, network 1004, and server 1005. Network 1004 is used as a medium to provide communication links between terminal devices 1001, 1002, and 1003 and server 1005. Network 1004 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0267] Users can use terminal devices 1001, 1002, and 1003 to interact with server 1005 via network 1004 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1001, 1002, and 1003, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0268] Terminal devices 1001, 1002, and 1003 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0269] Server 1005 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 1001, 1002, and 1003 (this is just an example). The backend management server can analyze and process data such as received item information query requests, and then feed the processing results back to the terminal devices.

[0270] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by server 1005, and correspondingly, the data processing device is generally set in server 1005.

[0271] It should be understood that Figure 10 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0272] The following is for reference. Figure 11 It shows a schematic diagram of the structure of a computer system 1100 suitable for implementing a terminal device of the present invention. Figure 11 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0273] like Figure 11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1102 or programs loaded from storage section 1108 into random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of the system 1100. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0274] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0275] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs the functions defined above in the system of this invention.

[0276] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0277] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0278] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a parsing module and a processing module, wherein the names of these modules do not necessarily limit the module itself.

[0279] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: receiving and parsing a business request to obtain dimension values ​​corresponding to each dimension and statistical parameter values ​​corresponding to each statistical parameter; inserting a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension and the statistical parameter values ​​corresponding to each statistical parameter as key-value pairs; wherein the multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, and the feature object defines each dimension and each statistical parameter.

[0280] In another aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0281] According to the technical solution of this invention, by using the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs to insert a multi-way balanced search tree, wherein the multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, the technical means overcome the technical problems of high memory consumption or non-trackable data in the prior art. This invention creates a multi-way balanced search tree based on the number of references of each dimension and inserts dimension values, statistical parameter values, and event time as key-value pairs, supporting data reuse from the bottom layer to save memory resources. Furthermore, the relevant data of the feature object is traceable, and the calculation results of the feature object are accurate.

[0282] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Receive and parse business requests to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time; among which, the dimensions and statistical parameters that need to be parsed from the business request are determined by the feature object. Based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, insert a multi-way balanced search tree; The multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, and the feature object defines each dimension and each statistical parameter. Using the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs, a multi-way balanced search tree is inserted, including: Determine the root node of the current dimension in a multi-way balanced search tree; In the multi-way balanced search tree, find the same source node of the root node, and use the dimension value of the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and insert it into the same source node; In the multi-way balanced search tree, find the heterogeneous node of the root node, and use the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the heterogeneous node.

2. The method according to claim 1, characterized in that, Before receiving and parsing business requests, the following steps are also included: The order of each dimension is determined based on the number of times each dimension is referenced in each feature object; Based on the order of the dimensions, a multi-way balanced search tree is constructed, such that heterogeneous nodes in the multi-way balanced search tree are mounted sequentially according to the order of the dimensions. In the multi-way balanced search tree, "same origin" means that the parent node and child node have the same dimension, while "different origin" means that the parent node and child node have different dimensions.

3. The method according to claim 2, characterized in that, The order of each dimension is determined based on the number of times each dimension is referenced in each feature object, including: A greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object from largest to smallest, thereby determining the order of the dimensions.

4. The method according to claim 3, characterized in that, A greedy optimization algorithm is used to sort the dimensions based on their number of references in each feature object, from highest to lowest, including: Step 41) Select the largest dimension from the list of feature objects that has not yet been optimized and has the highest number of references; Step 42) Based on the maximum dimension, divide the feature object list into a feature object sublist that contains the maximum dimension and a feature object sublist that does not contain the maximum dimension; Step 43) For the feature object sublist containing the maximum dimension, move the dimensions that are located before the maximum dimension and appear less frequently than the maximum dimension to the end of the maximum dimension, and mark the maximum dimension as optimized; Step 44) Repeat steps 41) to 43) until the largest dimension that has not been optimized and has the most references cannot be selected from the list of feature objects.

5. The method according to claim 1, characterized in that, In the multi-way balanced search tree, find the same-source node of the root node, and insert the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the same-source node, including: Step 71): Determine if the list of same-origin child nodes of the current node is empty; if yes, insert the dimension value corresponding to the dimension of the same-origin node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the current node; if no, proceed to step 72). Step 72): Using the dimension value corresponding to the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, determine whether the key-value pair to be inserted exists in the key-value pair list of the current node; if yes, directly return the current node; if no, proceed to step 73). Step 73) Based on the size relationship between the key-value pair to be inserted and the existing key-value pairs in the current node, continue to search for the node to be inserted from the same source left child node or same source right child node of the existing key-value pair in the current node, and then insert the key-value pair to be inserted into it.

6. The method according to claim 1, characterized in that, In the multi-way balanced search tree, the heterogeneous node of the root node is searched, and the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time are used as key-value pairs to be inserted into the heterogeneous node, including: Determine whether there exists a heterogeneous child node in the list of heterogeneous child nodes corresponding to the key-value pair to be inserted, with the same dimension as the dimension to which the key-value pair to be inserted belongs; If so, then the heterogeneous child node will be used as the new root node; If not, initialize a new heterogeneous child node, insert the new heterogeneous child node into the heterogeneous child node list of key-value pairs corresponding to the root node, so that each heterogeneous child node in the heterogeneous child node list is arranged in ascending order according to its corresponding dimension; and, use the new heterogeneous child node as the new root node. In the multi-way balanced search tree, find the same source node of the new root node, and use the dimension value corresponding to the dimension of the same source node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the same source node.

7. The method according to claim 1, characterized in that, After inserting the dimensional values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs into the multi-way balanced search tree, the method further includes: The number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the business requests are traced back to their source.

8. The method according to claim 7, characterized in that, The number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the data source of the business requests is traced, including: Locate the logical root node of the node containing the key-value pair corresponding to the statistical parameter value and the event time in the multi-way balanced search tree; The key-value pairs in the logical root node and its child nodes are traversed to calculate the number of business requests corresponding to each feature object, and the business requests are traced by dimension value.

9. The method according to claim 7, characterized in that, The number of business requests corresponding to each feature object is calculated based on the multi-way balanced search tree, and the data source of the business requests is traced, including: The number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced back to their dimensional values.

10. The method according to claim 9, characterized in that, The number of business requests corresponding to each feature object is calculated by performing a depth-first traversal of the key-value pairs in each node of the multi-way balanced search tree, and the business requests are traced back to their dimensional values, including: Step 121): Determine whether the current node corresponds to the statistical parameter value and the event time; if yes, proceed to step 122); if no, proceed to step 123). Step 122) Traverse the key-value pairs in the current node and its child nodes to calculate the number of business requests corresponding to each feature object, and trace the business requests by dimension value. Step 123): Recursively select the heterogeneous child nodes of the current node as the new current node and execute step 121.

11. A data processing apparatus, characterized in that, include: The parsing module is used to receive and parse business requests to obtain the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time; among them, the dimensions and statistical parameters that need to be parsed from the business request are determined by the feature object. The processing module is used to insert a multi-way balanced search tree based on the dimension values ​​corresponding to each dimension, the statistical parameter values ​​corresponding to each statistical parameter, and the event time as key-value pairs; The multi-way balanced search tree is constructed based on the number of times each dimension is referenced in each feature object, and the feature object defines each dimension and each statistical parameter. The processing module is also used to determine the root node of the current dimension in a multi-way balanced search tree; In the multi-way balanced search tree, find the same source node of the root node, and use the dimension value of the current dimension or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted, and insert it into the same source node; In the multi-way balanced search tree, find the heterogeneous node of the root node, and use the dimension value corresponding to the dimension of the heterogeneous node or the statistical value corresponding to each statistical parameter and the event time as the key-value pair to be inserted into the heterogeneous node.

12. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-10.

13. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images