Data processing methods, apparatuses, electronic devices, storage media and software products
By locating data buckets on the decision tree and randomly reading data from within the buckets, the problem of high filtering latency caused by full traversal is solved, enabling real-time filtering and multi-dimensional attribute combination filtering under massive data, thus improving filtering efficiency and scalability.
Patent Information
- Application Number
- CN202611336238.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-31
- Publication Date
- 2026-11-17
Smart Images

Figure CN123115490A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more particularly to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology
[0002] With the continuous expansion of internet business, data is exhibiting characteristics of massive volume, multidimensionality, and high dynamism. In internet business scenarios, such as game item drop scenarios, e-commerce product recommendations, and precise advertising, it is often necessary to perform real-time and fair random selection from hundreds of thousands or even millions of candidate datasets based on multi-dimensional attributes such as category, level, quality, and tags.
[0003] In related technologies, multidimensional random screening often employs a full-scale traversal retrieval approach. This method requires comparing each candidate data item in the entire dataset against the screening criteria before randomly selecting the matching data entries. However, this approach has significant drawbacks: a full traversal requires scanning all candidate data item by item to complete the condition comparison. The larger the total amount of data, the longer the screening process takes, resulting in a significant increase in system response latency and making it difficult to meet real-time screening performance requirements. Summary of the Invention
[0004] This disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product to solve the problems of high latency and inability to adapt to massive data in the full traversal scheme in related technologies, and to ensure the performance requirements of real-time filtering.
[0005] According to one aspect of this disclosure, a data processing method is provided, the method comprising: Receive a data filtering request, the data filtering request including data filtering conditions and the number of data to be extracted; The data filtering conditions are compiled to obtain decision tree traversal instructions, and the decision tree traversal instructions are executed on the pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions; wherein, the data decision tree includes a root node, intermediate nodes and a first data bucket containing stored business data, the root node is used to indicate the highest-level attribute dimension of the business data, the intermediate nodes are used to indicate the second-level attribute dimension of the business data, and the first data bucket contains the second data bucket; A third data bucket is determined from at least one second data bucket, and based on the business data stored in the third data bucket and the number of data extracted, target business data corresponding to the data filtering conditions is obtained, and the target business data is returned as a request response result corresponding to the data filtering request.
[0006] According to another aspect of this disclosure, a data processing apparatus is provided, the apparatus comprising: The first module is used to receive a data filtering request, which includes data filtering conditions and the number of data to be extracted. The second module is used to compile the data filtering conditions to obtain decision tree traversal instructions, and execute the decision tree traversal instructions on the pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions; wherein, the data decision tree includes a root node, intermediate nodes and a first data bucket containing stored business data, the root node is used to indicate the highest-level attribute dimension of the business data, the intermediate nodes are used to indicate the second-level attribute dimension of the business data, and the first data bucket contains the second data bucket; The third module is used to determine a third data bucket from at least one second data bucket, and based on the business data stored in the third data bucket and the number of data extracted, obtain target business data corresponding to the data filtering conditions, and return the target business data as a request response result corresponding to the data filtering request.
[0007] According to another aspect of this disclosure, an electronic device is provided, the electronic device comprising: One or more processors; Storage device for storing one or more programs. When one or more programs are executed by one or more processors, the one or more processors implement any of the data processing methods described in the embodiments of this disclosure.
[0008] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement any of the data processing methods described in the embodiments of this disclosure.
[0009] According to another aspect of the present disclosure, a computer program product is provided, which, when executed by a processor, implements any of the data processing methods described in the embodiments of the present disclosure.
[0010] The technical solution of this disclosure, by receiving standardized data filtering requests through a unified entry point, decouples the filtering requirements of the business side from the underlying filtering execution logic. Business parties only need to focus on their own business rules, without needing to be aware of the implementation details of the underlying filtering. The unified request entry point allows for centralized management of all filtering requests, which, compared to the highly coupled implementation of business logic and filtering logic in the full traversal approach, facilitates subsequent iteration and expansion of filtering capabilities. Furthermore, by compiling filtering conditions into decision tree traversal instructions, candidate data buckets are directly located on the tree index structure, eliminating the need to scan all candidate data one by one. This significantly reduces the time overhead of filtering and solves the problem of increased latency with larger data volumes in the full traversal approach. Based on the multi-dimensional index structure of the decision tree, it naturally supports combined filtering of multi-dimensional attributes. Compared to the method of comparing data item by item in full traversal, the filtering efficiency does not significantly decrease with the growth of data scale, meeting the real-time filtering requirements of massive amounts of data. Furthermore, by selecting data buckets first and then directly and randomly reading data within those buckets, the time spent on random extraction can be controlled within a fixed range, not increasing with the total amount of data, thus ensuring filtering performance in high-concurrency scenarios. The extraction method based on direct indexing within buckets eliminates the need for secondary traversal and comparison of data within buckets, significantly reducing the computational overhead of a single extraction compared to the full traversal scheme, and supporting millisecond-level real-time filtering requirements. The technical solution of this disclosure, by constructing a multi-dimensional decision tree to achieve bucket indexing of data, transforms the filtering process into decision tree path positioning and random reading within buckets. This effectively solves the problems of high latency and inability to adapt to massive amounts of data in related technologies using the full traversal scheme, ensuring the performance requirements of real-time filtering. Moreover, the tree structure-based multi-dimensional data indexing can adapt to the combined filtering needs of multiple dimensions and has good scalability.
[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 A schematic flowchart illustrating a data processing method provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating another data processing method provided in an embodiment of this disclosure; Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0014] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.
[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0017] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0018] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0019] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0021] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this disclosure. This embodiment is applicable to any scenario involving the filtering of business data. The method can be executed by a data processing device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers or servers. Figure 1 As shown, the method in this embodiment includes: S110. Receive a data filtering request, which includes data filtering conditions and the number of data to be extracted.
[0022] A data filtering request can be understood as a service request initiated by the business side to trigger the data filtering process. A data filtering request can be generated by an upstream business system, carrying instructions with filtering rules and extraction scale parameters. A data filtering request can be used to obtain business data that meets specified conditions from the filtering system. For example, a call instruction initiated by the game business side to "obtain 3 items that meet specified conditions" is a data filtering request.
[0023] In one scenario, a data filtering request may include data filtering criteria and the number of data to be extracted. Data filtering criteria can be understood as a set of attribute constraint rules used to limit the range of candidate data. Data filtering criteria can be rules that limit the values and ranges of multiple attribute dimensions of business data. Data filtering criteria can be used to define the range of candidate data that can participate in data extraction. For example, data filtering criteria could be "category: weapon, level: 30 or higher, quality: epic," and business data that meets all three attribute requirements can enter the candidate data range.
[0024] The data extraction quantity can be understood as the total number of target business data entries that a single data filtering request needs to return. The data extraction quantity can be the number of data entries specified by the business side for the final output of this data filtering, used to control the number of data extraction executions and the scale of the results. For example, if the data filtering request specifies a data extraction quantity of 5, it means that this filtering ultimately needs to return 5 pieces of business data that meet the conditions.
[0025] In one scenario, a data filtering request initiated by an upstream business template can be received through a pre-defined service interface. The parameters in the data filtering request are parsed to extract the data filtering conditions used to limit the range of candidate data, as well as the number of data to be extracted in this filtering process. Furthermore, after verifying the validity of the parameter format and values, the parsed parameters are passed to subsequent processing steps to initiate the data filtering process.
[0026] S120. Compile the data filtering conditions to obtain decision tree traversal instructions, and execute the decision tree traversal instructions on the pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions.
[0027] In this context, decision tree traversal instructions can be understood as path matching instructions that can be executed on a data decision tree after transforming data filtering conditions. These instructions are generated after rule compilation and correspond to the decision logic of each level of the decision tree. Decision tree traversal instructions can be used to guide the decision tree traversal process, skipping branches that do not meet the conditions and quickly locating the target data range. For example, the data filtering condition "Category = Weapon, Quality = Epic" can be compiled into a traversal execution instruction that "matches the weapon branch at the root node, skips leaf nodes whose quality is not epic."
[0028] In this context, a data decision tree can be understood as a tree-shaped index structure built based on business attribute dimensions, used for fast indexing and bucketed storage of business data. A data decision tree can be a multi-dimensional data index structure organized according to attribute hierarchy, consisting of a root node, intermediate nodes, and leaf nodes that hold data buckets. Data decision trees can be used to quickly narrow down the range of candidate data and locate data sets that meet certain conditions. For example, a decision tree built according to a three-level attribute hierarchy of "category-level-quality" can achieve fast indexing of items with different attribute combinations.
[0029] In one scenario, a data decision tree may include a root node, intermediate nodes, leaf nodes, and a first data bucket mounted on the leaf nodes.
[0030] The root node can be understood as the starting decision node at the top level of the data decision tree. The root node corresponds to the highest-level and most core attribute dimension of the business data, serving as the unified entry point for traversing the data decision tree. All business data is first divided into its first-level scope based on this attribute dimension, extending downwards to branches corresponding to different attribute values. For example, the "Item Category" node at the top level of the data decision tree is the root node. The highest-level attribute dimension can be understood as the core attribute for the first-level, coarsest-grained division of all business data, and is the highest-priority division criterion in the data decision tree's hierarchical system. The highest-level attribute dimension can be the most powerful and fundamental attribute dimension among all filtering dimensions. As the first-level division standard of the data decision tree, the highest-level attribute dimension can split all candidate data from the source into several mutually exclusive categories, minimizing the range of data for subsequent filtering. When all business data enters the data decision tree, it first completes its initial classification based on this attribute dimension, forming the foundation for all subdivision levels. For example, in the game item selection scenario, if "item category" is set as the highest level attribute dimension, all items will be first divided into non-overlapping categories such as weapons, armor, and consumables, completing the initial sorting at the top level.
[0031] In a data decision tree, intermediate nodes can be understood as hierarchical decision nodes located between the root node and the underlying data containers. Intermediate nodes can indicate the secondary attribute dimensions of business data. They are used to further refine the data scope based on the top-level division. For example, the "Item Level" node below the root node is an intermediate node. Depending on the number of secondary attribute dimensions, intermediate nodes can be set to one or more layers, progressively narrowing the data filtering range. Secondary attribute dimensions can be understood as attributes that further refine the data based on the highest-level division, with a lower priority than the highest level, corresponding to the middle layers of the data decision tree. Secondary attribute dimensions can be used to further narrow the data scope given that the top-level classification has already been determined. They can be set to one or more layers, with lower levels having finer granularity. Each level of attribute dimension only further subdivides within the results of the previous level, gradually converging the data scope into a smaller set. For example, after the top-level division of "item category", "item level" is the next level attribute dimension. Under the major category of weapons, it is further subdivided according to different level ranges.
[0032] In this context, leaf nodes can be understood as the lowest-level, terminal nodes in the data decision tree, representing the endpoint of hierarchical decision-making. Leaf nodes correspond to the finest-grained attribute division dimensions. Each leaf node can be associated with a corresponding first data bucket; business data satisfying the full-path attribute conditions of that leaf node is stored in its corresponding first data bucket. For example, the lowest-level "Item Quality" node is a leaf node. Leaf nodes can also indicate the lowest-level attribute dimension of business data. The lowest-level attribute dimension can be understood as the finest-grained, terminal division attribute in the decision tree hierarchy. This lowest-level attribute dimension can be the final dimension for hierarchical decision-making, corresponding to the decision rules of the leaf nodes. After being divided by this attribute dimension, the business data is finally categorized and stored in its corresponding first data bucket. For example, "Item Quality," under the "Item Level" dimension, is the lowest-level attribute dimension.
[0033] The first data bucket can be understood as a storage unit mounted on a leaf node of the data decision tree, used to store actual business data. The first data bucket can correspond to a fixed combination of attributes along a complete path in the data decision tree; all business data satisfying all attribute conditions of that path will be stored in the corresponding first data bucket. The first data bucket can be the carrier of business data in the decision tree structure. For example, the leaf node container corresponding to the decision tree path "weapon type, level greater than or equal to 30, epic quality" is a first data bucket, storing all game item data that matches this attribute combination. The first data bucket can contain a second data bucket.
[0034] The second data bucket can be understood as the leaf node data buckets that meet the current data filtering conditions after being filtered by the decision tree path. The second data bucket can be the set of all first data buckets that meet the filtering conditions after executing the decision tree traversal instruction; it represents the candidate data range corresponding to this data filtering request. For example, when the data filtering condition is "weapon, epic quality," all first data buckets with the category "weapon" and quality "epic" will be hit, and these hit first data buckets are the second data buckets.
[0035] In one scenario, the provided solution compiles data filtering conditions to obtain decision tree traversal instructions, which may include at least one of the following: processing the data filtering conditions using a preset instruction template to obtain decision tree traversal instructions; performing syntax parsing on the logical expressions corresponding to the data filtering conditions to generate an abstract syntax structure containing dimension decision rules and logical relationships, then mapping each dimension decision node in the syntax structure to the hierarchical nodes of the decision tree one by one, converting them into branch decision and jump instructions for the corresponding level, and finally combining them into complete decision tree traversal instructions; or using a built-in rule engine to decompose the data filtering conditions into unit decision rules corresponding to each attribute dimension, then sorting the unit decision rules according to the node hierarchy of the data decision tree, generating branch selection and skip instructions for each level of nodes in turn, and finally splicing them together to form complete decision tree traversal instructions. The following is a detailed explanation of one of the instruction generation methods.
[0036] Optionally, the data filtering conditions are compiled to obtain decision tree traversal instructions, including: processing the data filtering conditions using a preset instruction template to obtain decision tree traversal instructions.
[0037] The preset instruction template can be understood as a standardized template carrier that is pre-configured and used to transform data filtering conditions into decision tree traversal instructions. The preset instruction template can be a predefined fixed sentence structure and placeholder framework, with reserved spaces inside the framework for filling in actual attributes and attribute values. The preset instruction template can specify the output format, node jump syntax, and branch filtering syntax of the decision tree traversal instructions. It eliminates the need to construct decision tree traversal instructions from scratch each time; simply filling the parameters from the data filtering conditions into the corresponding positions in the preset instruction template will output instructions that conform to the format recognizable by the data decision tree.
[0038] In one scenario, a pre-stored preset instruction template is read, the data filtering conditions are parsed, and attribute dimensions, comparison operators, and attribute values are extracted. These extracted parameters are then filled into the corresponding placeholders in the preset instruction template, completing the conversion from conditions to instruction format. This generates a decision tree traversal instruction that the data decision tree can recognize and execute. Further, the generated decision tree traversal instruction is read, starting from the root node of the pre-constructed data decision tree. Matching is performed layer by layer downwards according to the judgment requirements of each level node in the decision tree traversal instruction, skipping all branches that do not meet the judgment requirements. The first data buckets corresponding to the leaf nodes hit during the traversal are continuously collected, and finally one or more successfully matched first data buckets are output, resulting in at least one second data bucket corresponding to the data filtering conditions.
[0039] S130. Determine a third data bucket from at least one second data bucket, and based on the business data stored in the third data bucket and the number of data extracted, obtain the target business data corresponding to the data filtering conditions, and return the target business data as the request response result corresponding to the data filtering request.
[0040] The third data bucket can be understood as a single data storage container selected from the candidate set of second data buckets for this data extraction. The third data bucket may be the final extraction carrier determined by the bucket selection logic, and the target business data required for this screening is obtained from this bucket.
[0041] In this context, business data can be understood as actual business entity data with multi-dimensional attributes stored in data buckets. Business data serves as the processing object of the entire filtering process. Each piece of business data carries multiple different business attribute fields, which can correspond to the partitioning dimensions of each level of the data decision tree. Business data can be allocated to the corresponding data bucket for storage according to the decision tree path corresponding to its attributes. For example, a single piece of game item information containing attributes such as item identifier, category, level, and quality constitutes a piece of business data.
[0042] In this context, target business data can be understood as business data entries that meet all constraints after filtering and data extraction. Target business data can be business data extracted from the third data bucket that meets the data filtering criteria, and it is the final output of this data filtering process. Target business data can be used to respond to data filtering requests from the business side. For example, three epic-quality weapons / items randomly selected from the third data bucket would be the target business data for this filtering process.
[0043] The request response result can be understood as a standardized processing result carrier returned to the request initiator after the filtering process is completed. The request response result can encapsulate the target business data obtained in this filtering, organized according to the agreed interface format, and is used to feed the filtering results back to the upstream business system, marking the completion of this filtering request processing.
[0044] In one scenario, the number of second data buckets may include one or more. For different numbers of second data buckets, there may be different ways to determine the third data bucket. The methods for determining the third data bucket in these two scenarios will be explained below.
[0045] Optionally, determining a third data bucket from at least one second data bucket includes: determining a second data bucket as a third data bucket in response to the number of second data buckets being one.
[0046] In one scenario, if the number of second data buckets obtained is one, that second data bucket can be identified as the third data bucket.
[0047] Optionally, determining a third data bucket from at least one second data bucket includes: in response to the existence of multiple second data buckets, determining a third data bucket from multiple second data buckets based on business data stored within the multiple second data buckets.
[0048] In one scenario, the provided solution determines a third data bucket based on the business data stored in multiple second data buckets. This can include at least one of the following: performing uniform random sampling on the multiple second data buckets and selecting the randomly selected second data bucket as the third data bucket; pre-configuring selection priorities for data buckets corresponding to different attribute values, and after obtaining multiple second data buckets, comparing the priority levels of each second data bucket and selecting the second data bucket with the highest priority as the third data bucket; determining the corresponding sampling probability based on the amount of business data in each second data bucket, and then sampling the multiple second data buckets according to this sampling probability, selecting the selected second data bucket as the third data bucket. The following provides a detailed explanation of one method for determining the third data bucket.
[0049] Optionally, a third data bucket is determined from the multiple second data buckets based on the business data stored in the multiple second data buckets, including: for the multiple second data buckets, determining the sampling probability corresponding to the second data bucket based on the total amount of business data stored in the second data bucket; sampling the multiple second data buckets based on the sampling probabilities corresponding to the multiple second data buckets respectively, and determining the sampled second data bucket as the third data bucket.
[0050] The total number of business data items can be the total number of business data entries actually stored within a single second data bucket. Alternatively, the total number of business data items can be a metric measuring the data size of each second data bucket, representing the total amount of business data stored within that bucket. For example, if a second data bucket stores 200 records of epic weapon items, then 200 represents the total number of business data items corresponding to that second data bucket.
[0051] The sampling probability can be understood as the likelihood of each second data bucket being selected during the random selection process among buckets. The sampling probability is calculated based on the amount of business data in each second data bucket. The proportion of data in a single bucket to the total data in all second data buckets is the sampling probability for that second data bucket; the larger the data volume of a bucket, the higher the sampling probability. For example, if three second data buckets have 200, 300, and 500 data entries respectively, with a total data volume of 1000, then the sampling probabilities for the three buckets are 20%, 30%, and 50%, respectively.
[0052] Sampling can be an operation that randomly selects a target object from a candidate set according to a preset probability rule. Sampling can be a process of randomly selecting a single target data bucket within a candidate range consisting of multiple second data buckets, based on the sampling probability corresponding to each bucket, and finally selecting a unique target data bucket. This is the core execution step for implementing weighted random bucket selection.
[0053] In one scenario, the total amount of business data stored in each second data bucket is obtained sequentially. Using the sum of the total amount of business data in all second data buckets as the calculation base, the data volume ratio of each second data bucket is calculated. This ratio is used as the sampling probability of the corresponding second data bucket. Then, weighted random sampling is performed according to the sampling probability of each second data bucket to randomly select one from multiple candidate second data buckets, and the selected second data bucket is determined as the third data bucket.
[0054] In one scenario, the provided solution obtains the target business data corresponding to the data filtering conditions based on the business data stored in the third data bucket and the number of data extracted. This can include at least one of the following: generating a random index with the total number of business data stored in the third data bucket as the upper limit; directly reading the corresponding business data based on the random index; repeatedly performing the above operation according to the number of data extracted to determine the read business data as the target business data corresponding to the data filtering conditions; generating a set of random indexes corresponding to the number of data extracted at once; then accessing the underlying storage location of the third data bucket in batches based on the index set; reading all the business data at the corresponding locations at once to obtain the target business data; performing a random shuffling operation on the full data index corresponding to the third data bucket to obtain a disordered index sequence; then, according to the number of data extracted, extracting a corresponding number of indexes from the beginning of the disordered sequence; and reading the business data at the corresponding locations as the target business data. The following provides a detailed explanation of one of these data extraction methods.
[0055] Optionally, based on the business data stored in the third data bucket and the number of data extracted, the target business data corresponding to the data filtering conditions is obtained, including: using the total number of business data stored in the third data bucket as the upper limit of the value, and using a random generator to generate a random index; directly accessing the third data bucket according to the random index and reading the business data corresponding to the random index; cyclically executing the random index generation and business data reading operations according to the number of data extracted, and determining the read business data as the target business data corresponding to the data filtering conditions.
[0056] The upper limit of the value can be understood as the maximum numerical boundary allowed when generating a random index. This upper limit can constrain the output range of the random generator, ensuring that the generated index value does not exceed the actual total amount of data in the third data bucket, thus avoiding reading non-existent or invalid storage locations.
[0057] A random number generator can be a functional component used to generate random values within a specified range. It is the core execution unit for random selection, ensuring the randomness and fairness of the selection results. For example, a built-in random number generation component that generates random integers within a specified range is a random number generator.
[0058] The random index can be a randomly generated location identifier corresponding to the storage location of business data within a data bucket. The random index can be the location number corresponding to each piece of business data within the data bucket. By using the randomly generated index value, the corresponding data entry can be directly located without needing to traverse each entry individually. For example, if the generated random value is 126, this value is the random index, corresponding to the 126th piece of business data stored in the third data bucket.
[0059] Direct access can be understood as a data reading method that directly locates and reads data using a random index. Direct access allows data to be read directly from the corresponding storage location based on the index value without scanning the data in the bucket one by one. The reading time is fixed and is not affected by the growth of the total amount of data.
[0060] In one scenario, the total amount of business data stored in the third data bucket can be used as the upper limit to generate a random index within the valid range using the system's built-in random generator. Furthermore, the corresponding storage location in the third data bucket can be directly located and accessed based on this random index, and the business data corresponding to the indexed location can be read. Alternatively, the random index generation and business data reading operations can be performed cyclically according to the number of data extractions. The loop stops after the required number of data extractions has been read, and all read business data is identified as the target business data corresponding to the data filtering conditions.
[0061] In one scenario, the provided solution supports configuring multiple data sampling modes during the extraction of target business data based on a third data bucket. This is to adapt to the varying requirements for sampling result repeatability across different business scenarios. The solution also executes the corresponding extraction control logic based on the currently active sampling mode. For non-replacement sampling scenarios where single extraction results must be unique, to prevent repeated hits on the same business data during cyclic sampling and ensure the uniqueness of the extraction results, a corresponding marking process can be performed: in response to the non-replacement sampling mode, a selection mark is added to the read business data.
[0062] In this context, the data sampling mode can be understood as a predefined operating mode used to constrain the rules for repeated data extraction. The data sampling mode can be a system of rules specifying whether a single piece of business data can be selected repeatedly within the same extraction process; it is a configurable item in the extraction logic. The data sampling mode can be flexibly switched according to business needs and can include both sampling with replacement and sampling without replacement.
[0063] The sampling mode without replacement can be understood as a sampling mode in which a single piece of business data can only be selected once in a single sampling process. The sampling mode without replacement can be a type of data sampling mode. Its core rule is that business data that has been read and selected will be removed from the candidate sampling pool and will not participate in random selection in subsequent loop sampling. The final output target business data are all independent entries that do not repeat.
[0064] The selection marker can be a temporary status identifier added to the extracted business data. The selection marker is used to indicate that the data has been selected. When generating random indexes to read data subsequently, data with this marker will be automatically skipped, thus preventing duplicate extraction. The selection marker can be cleared after the extraction process is complete, restoring the original state of the business data. For example, after a weapon item data is read, a temporary "extracted" marker can be added to it, so that this item data will not be selected again during subsequent loop extractions.
[0065] In one scenario, during the execution of extracting target business data based on the third data bucket, in response to the data sampling mode being non-replacement sampling mode, if business data is read from the third data bucket according to the random index, a selection status flag can be added to the business data read this time to prevent the same business data from being repeatedly hit during subsequent loop extractions.
[0066] The technical solution of this disclosure, by receiving standardized data filtering requests through a unified entry point, decouples the filtering requirements of the business side from the underlying filtering execution logic. Business parties only need to focus on their own business rules, without needing to be aware of the implementation details of the underlying filtering. The unified request entry point allows for centralized management of all filtering requests, which, compared to the highly coupled implementation of business logic and filtering logic in the full traversal approach, facilitates subsequent iteration and expansion of filtering capabilities. Furthermore, by compiling filtering conditions into decision tree traversal instructions, candidate data buckets are directly located on the tree index structure, eliminating the need to scan all candidate data one by one. This significantly reduces the time overhead of filtering and solves the problem of increased latency with larger data volumes in the full traversal approach. Based on the multi-dimensional index structure of the decision tree, it naturally supports combined filtering of multi-dimensional attributes. Compared to the method of comparing data item by item in full traversal, the filtering efficiency does not significantly decrease with the growth of data scale, meeting the real-time filtering requirements of massive amounts of data. Furthermore, by selecting data buckets first and then directly and randomly reading data within those buckets, the time spent on random extraction can be controlled within a fixed range, not increasing with the total amount of data, thus ensuring filtering performance in high-concurrency scenarios. The extraction method based on direct indexing within buckets eliminates the need for secondary traversal and comparison of data within buckets, significantly reducing the computational overhead of a single extraction compared to the full traversal scheme, and supporting millisecond-level real-time filtering requirements. The technical solution of this disclosure, by constructing a multi-dimensional decision tree to achieve bucket indexing of data, transforms the filtering process into decision tree path positioning and random reading within buckets. This effectively solves the problems of high latency and inability to adapt to massive amounts of data in related technologies using the full traversal scheme, ensuring the performance requirements of real-time filtering. Moreover, the tree structure-based multi-dimensional data indexing can adapt to the combined filtering needs of multiple dimensions and has good scalability.
[0067] Figure 2 This is a schematic flowchart illustrating another data processing method provided in this embodiment. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 2 As shown, the method in this embodiment may specifically include: S210. Obtain multidimensional data stream and preset data filtering rules. The multidimensional data stream contains multiple business data items, each of which carries multiple business attribute fields of different levels and dimensions.
[0068] In this context, a multidimensional data stream can be understood as a batch input data stream carrying multiple business data items with multidimensional attributes. A multidimensional data stream can be the original data source for constructing a data decision tree. It consists of multiple independent business data items arranged in an ordered manner, each carrying multiple attribute information belonging to different levels. This data stream is fed into the construction process as input, providing the decision tree with all the business data to be classified. For example, a batch import of a full dataset of game items, where each item has multiple attributes such as category, level, and quality, inputting it into the construction system in the form of a stream, constitutes a multidimensional data stream.
[0069] In this context, business data can be understood as entity data entries with multi-dimensional business attributes that need to be categorized and stored according to rules. Business data can be the core processing object in the decision tree construction process. Each piece of business data corresponds to a business entity and carries multiple business attribute fields at different levels. The system will match the corresponding path in the decision tree based on its attribute values and ultimately store it in the corresponding data bucket. For example, a single piece of game item data containing item identifier, category, level, and quality information is a piece of business data.
[0070] In this context, attribute dimensions can be understood as attribute categories used to classify and categorize business data. Attribute dimensions can serve as the baseline dimensions for hierarchical classification of business data, divided into different levels according to the granularity of classification, from coarse to fine. Higher levels correspond to coarse-grained broad categories, while lower levels correspond to fine-grained subcategories, corresponding one-to-one with the root node, intermediate nodes, and leaf nodes of a decision tree, thus forming a multi-level classification system. For example, item category, item level, and item quality are three different levels of attribute dimensions.
[0071] In this context, business attribute fields can be understood as field units in business data that record specific attribute values. A business attribute field is a concrete representation of an attribute dimension on a single piece of business data; each field corresponds to an attribute dimension, recording the specific value of the data under that dimension. The system determines the decision tree branch and target data bucket to which the business data should belong by reading the values of these fields. For example, the specific field content of "Category = Weapon," "Level = 35," and "Quality = Epic" in a piece of item data are the corresponding business attribute fields.
[0072] The preset data filtering rules can be understood as a set of pre-defined classification rules used to define the hierarchical division logic of the decision tree. These preset data filtering rules serve as the core basis for building the decision tree structure, clearly specifying the number of levels, the attribute dimensions corresponding to each level, and the branching rules for each dimension. They determine the classification granularity and node arrangement of the entire tree, ensuring that the constructed decision tree meets the business's classification query requirements. For example, a pre-defined rule that "divides the tree into three levels sequentially based on item category, item level, and item quality" is a preset data filtering rule.
[0073] In one scenario, when initiating the full construction task of the data decision tree, the first step is to obtain two types of core input resources required for this construction: one is a multi-dimensional data stream carrying all the business data to be classified. This data stream contains multiple independent business data, and each business data carries business attribute fields corresponding to different hierarchical attribute dimensions, which serve as the original data objects for subsequent classification and storage; the other is a pre-configured preset data filtering rule, which is used to clarify the hierarchical division dimensions and branch judgment logic of the decision tree, serving as the benchmark for subsequent construction of the tree structure.
[0074] S220. Based on the preset data filtering rules, initialize and construct an empty decision tree; the empty decision tree includes the root node, intermediate nodes, and the fourth data bucket for the business data to be written. The root node, intermediate nodes, and leaf nodes correspond to different levels of attribute dimensions.
[0075] Initialization construction can be understood as the process of building the basic framework of a decision tree without data, based on classification rules. Initialization construction can be the first step in decision tree construction, where the hierarchical structure of the tree nodes is built solely according to preset data filtering rules, generating the corresponding number of root nodes, intermediate nodes, and leaf nodes, and creating an empty data bucket for each leaf node. This stage only builds the structural framework and does not write any actual business data. For example, based on a three-level classification rule, a complete tree framework consisting of a root node, two levels of intermediate nodes, all leaf nodes, and corresponding empty data buckets is first generated; this construction process is called initialization construction.
[0076] An empty decision tree can be understood as an initial decision tree framework that has only completed the node structure construction but has not yet written business data. An empty decision tree can be the output of the initial construction, with a complete hierarchical node structure and branching logic, consisting of a root node, intermediate nodes, leaf nodes, and an empty fourth data bucket mounted on the leaf nodes; all data buckets are in an empty state and do not yet have data query capabilities. Once data is written, it can become a usable data decision tree.
[0077] The fourth data bucket can be understood as an empty storage container attached to an empty decision tree leaf node, containing business data to be written. The fourth data bucket can be the initial empty state of the first data bucket, generated synchronously with the leaf nodes during the decision tree initialization phase. At this time, no business data has been stored, and it is specifically used to receive subsequent business data written according to attribute categories. After the data writing is completed, it becomes the first data bucket carrying the business data. For example, an empty storage unit attached to an empty decision tree leaf node that has not yet contained item data is the fourth data bucket.
[0078] In one scenario, the pre-defined data filtering rules are first parsed to extract the number of attribute levels, the attribute dimensions corresponding to each level, and the branching rules for each dimension, thus clarifying the structural parameters of the entire decision tree. Then, in order from top to bottom, the root node corresponding to the highest level attribute dimension, the intermediate nodes corresponding to each next higher level attribute dimension, and the leaf nodes corresponding to the lowest level attribute dimension are generated sequentially. A fourth data bucket for writing business data is also generated for each leaf node, thereby completing the initial construction of an empty decision tree without actual business data.
[0079] S230. Determine the corresponding fourth data bucket for multiple business data according to their respective business attribute fields, and write the business data into the corresponding fourth data bucket to obtain a data decision tree.
[0080] In one scenario, all business data in the multidimensional data stream is traversed one by one. The business attribute fields of each level carried by each business data are parsed and their values are obtained. Starting from the root node of the initialized empty decision tree, the corresponding branch nodes are matched layer by layer according to the hierarchical order of the attribute dimensions. The process is traversed downwards along the decision path until the target leaf node at the end is located, and then the fourth data bucket attached to the leaf node is determined. Subsequently, the current business data is written into the corresponding fourth data bucket to complete the classification and storage. After all business data has completed path matching and writing and storage, the originally empty fourth data buckets are filled with business data of the corresponding attributes, and finally a data decision tree with complete structure and data is obtained.
[0081] S240. Receive a data filtering request. The data filtering request includes data filtering conditions and the number of data to be extracted.
[0082] S250. Compile the data filtering conditions to obtain decision tree traversal instructions, and execute the decision tree traversal instructions on the pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions.
[0083] S260. Determine a third data bucket from at least one second data bucket, and based on the business data stored in the third data bucket and the number of data extracted, obtain the target business data corresponding to the data filtering conditions, and return the target business data as the request response result corresponding to the data filtering request.
[0084] The technical solution of this disclosure first initializes and builds an empty decision tree framework of corresponding hierarchical dimensions based on preset data filtering rules, and then writes business data carrying multi-level attribute fields into corresponding data buckets according to attribute matching. This enables the hierarchical and standardized collection and storage of all business data according to attribute dimensions, constructing a data decision tree with a well-structured and clearly indexed structure. It can significantly reduce the matching path of multi-dimensional condition filtering by relying on the tree-like hierarchical structure, effectively improving the execution efficiency of subsequent data filtering and random sampling. It can also ensure that the data distribution and attribute classification logic correspond one-to-one, ensuring the accuracy of data classification. At the same time, the well-structured bucket storage structure also provides unified structural support for subsequent storage optimization operations such as load balancing verification, data migration and bucket merging, reducing the data management and scheduling costs throughout the entire lifecycle.
[0085] Figure 3 This is a schematic flowchart illustrating another data processing method provided in this embodiment. The technical solution of this embodiment can be combined with other embodiments; for the same or related parts, they can be described in conjunction with the descriptions of other embodiments, and will not be repeated here. Figure 3 As shown, the method in this embodiment may specifically include: S310. Receive a data filtering request. The data filtering request includes data filtering conditions and the number of data to be extracted.
[0086] S320. Compile the data filtering conditions to obtain decision tree traversal instructions, and execute the decision tree traversal instructions on the pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions.
[0087] S330. Determine a third data bucket from at least one second data bucket, and based on the business data stored in the third data bucket and the number of data extracted, obtain the target business data corresponding to the data filtering conditions, and return the target business data as the request response result corresponding to the data filtering request.
[0088] S340. For multiple first data buckets associated with intermediate nodes in the data decision tree, obtain the total amount of business data stored in the first data bucket.
[0089] In this context, the multiple first data buckets associated with an intermediate node can be understood as all the first data buckets attached to all lower-level leaf nodes under the same intermediate node's jurisdiction. An intermediate node extends downwards through multiple sub-branches, ultimately corresponding to multiple terminal leaf nodes, each attaching one first data bucket. These first data buckets all belong to the branch scope of that intermediate node and are the actual storage units for all business data under that intermediate node, collectively forming the intermediate node's total data pool. For example, an intermediate node corresponding to the dimension judgment of "level ≥ 30" extends to three leaf nodes: Common Quality, Rare Quality, and Epic Quality, each attaching three first data buckets. These three first data buckets are the multiple first data buckets associated with that intermediate node.
[0090] In one scenario, load inspection can be performed on the data decision tree. When performing periodic load inspection on the data decision tree, the target intermediate node to be inspected is first located. All leaf nodes within its jurisdiction are traversed along the lower branch path of the intermediate node. The first data buckets attached to each leaf node are obtained one by one. Then, the total number of business data entries stored in each first data bucket is counted. Thus, the total number of business data in each of the first data buckets associated with the intermediate node is obtained.
[0091] S350. Determine the load difference ratio corresponding to the intermediate node based on the total number of the multiple first data buckets.
[0092] The load variance ratio can be understood as a metric used to quantify the degree of data imbalance among multiple first data buckets under the same intermediate node. The load variance ratio can be calculated based on the total amount of business data in all first data buckets under the same intermediate node, and can be obtained using methods such as the ratio of the largest bucket's data volume to the smallest bucket's data volume, or the coefficient of variation of the data volume in each bucket. A higher load variance ratio indicates a more uneven distribution of data volume among the data buckets and a greater load difference.
[0093] Optionally, the load difference ratio corresponding to the leaf node is determined based on the total number corresponding to the multiple first data buckets, including: determining the minimum and maximum number from the total number corresponding to the multiple first data buckets; and determining the ratio between the maximum and minimum number to obtain the load difference ratio corresponding to the leaf node.
[0094] The minimum quantity can be understood as the smallest value among the total amount of business data in each of the first data buckets in the same group. The minimum quantity can be the smallest value selected after statistically analyzing the load scale of multiple first data buckets. It represents the load level of the first data bucket with the least data carrying capacity in the group and can be used as a lower limit benchmark for measuring load differences.
[0095] The maximum quantity can be understood as the largest value among the total amount of business data in each of the first data buckets in the same group. The maximum quantity can be the largest value selected after statistically analyzing the load scale of multiple first data buckets, representing the load level of the first data bucket with the largest data carrying capacity in the group, and can be used as an upper limit benchmark for measuring load differences.
[0096] In one scenario, for all the first data buckets attached to each leaf node under the same intermediate node, the total number of business data corresponding to each bucket is numerically traversed and compared. The minimum value of the smallest number and the maximum value of the largest number are extracted. Then, the ratio of the maximum value to the minimum value is calculated, and the obtained ratio is used as the load difference ratio corresponding to the leaf node group.
[0097] S360 processes business data stored in multiple first data buckets based on the load difference ratio.
[0098] The processing of business data stored in multiple first data buckets can be understood as a load balancing adjustment operation performed on the business data in each data bucket under the same intermediate node based on the degree of load difference. When the load difference ratio exceeds the ratio threshold, the data capacity of each first data bucket can be adjusted through methods such as data migration across buckets, data bucket splitting, and data bucket merging, thereby reducing the load gap between buckets and making the business data more evenly distributed, thus improving the efficiency and stability of subsequent data filtering and extraction.
[0099] Optionally, the business data stored in multiple first data buckets is processed based on the load difference ratio, including: maintaining state detection of the business data stored in multiple first data buckets in response to the load difference ratio being less than a ratio threshold; and performing data migration processing on the business data stored in multiple first data buckets in response to the load difference ratio not being less than a ratio threshold.
[0100] The ratio threshold can be understood as a pre-set critical value used to determine whether the load imbalance exceeds an acceptable range. The ratio threshold can serve as the dividing line for triggering load balancing operations and can be configured in advance according to business performance requirements. When the load difference ratio is less than this ratio threshold, it indicates that the imbalance in data distribution is within a reasonable range and no intervention is needed; otherwise, it is determined to be a load imbalance, requiring adjustment measures to be initiated.
[0101] Data migration processing can be a balancing operation that moves business data between multiple first data buckets to reduce the load difference between buckets. Data migration processing can be initiated when the load difference ratio exceeds a threshold. It can also be used to migrate some business data from a first data bucket with a larger data volume to a first data bucket with a smaller data volume, so that the data volume of each bucket tends to be balanced, reducing the load difference ratio and improving the stability and efficiency of overall data access.
[0102] In one scenario, the calculated load difference ratio is compared with a preset ratio threshold. If the load difference ratio is less than the threshold, it indicates that the load difference between the first data buckets under the same intermediate node is within an acceptable and reasonable range. In this case, routine status monitoring of the business data stored in the multiple first data buckets is maintained, and the data volume fluctuation of each bucket is continuously tracked. If the load difference ratio is not less than the threshold, it indicates that the load imbalance between buckets has exceeded the preset tolerance range. In this case, data migration processing is performed on the business data stored in the multiple first data buckets to balance the load scale of each bucket through data cross-bucket movement.
[0103] Optionally, data migration processing is performed on the business data stored in multiple first data buckets, including: receiving a data balancing instruction, which includes the first bucket identifier of the source data bucket, the second bucket identifier of the target data bucket, and a data migration strategy; determining a data migration path based on the data migration strategy, the positional relationship between the source data bucket and the target data bucket in the data decision tree; and migrating a portion of the business data in the source data bucket to the target data bucket based on the data migration path.
[0104] The data balancing command can be understood as a scheduling and control command used to trigger load balancing adjustments between data buckets. It serves as the entry point for initiating the data migration process and can be generated and issued after determining an inter-bucket load imbalance. The data balancing command encapsulates all the core parameters of the migration operation, clarifying the participating objects and execution rules to ensure the migration operation is executed in an orderly manner according to the expected goals. Optionally, the data balancing command may include the first bucket identifier of the source data bucket, the second bucket identifier of the target data bucket, and the data migration strategy.
[0105] In this context, the source data bucket can be understood as the first data bucket responsible for moving business data out during the data migration operation. The source data bucket can be a larger, more heavily loaded data bucket within the same branch, serving as the data supplier for this migration. After some business data is moved out of the source data bucket according to the data migration strategy, its own load will decrease accordingly, thereby narrowing the load difference between data buckets in the same group. The first bucket identifier can be a unique identifier used to identify the source data bucket. This identifier can also be a globally unique identification number for each data bucket, carried in the data balancing instruction, used to accurately locate the source container from which the data is moved out, ensuring accurate data source location during the migration operation and avoiding data offset.
[0106] In this data migration operation, the target data bucket can be understood as the first data bucket responsible for migrating business data. The target data bucket can be a smaller, less loaded data bucket within the same branch, serving as the data receiving end for this migration. After receiving the business data migrated from the source data bucket, the target data bucket's own load will increase accordingly, ultimately leading to a more even load distribution among the data buckets in the same group. The second bucket identifier can be understood as a unique identifier used to identify the target data bucket. This identifier can be a globally unique identification number corresponding to the target data bucket, carried in the data balancing instruction, used to accurately locate the target container for data migration, ensuring that business data moved from the source data bucket is accurately stored in the specified target data bucket.
[0107] In this context, a data migration strategy can be understood as an execution plan that describes the rules for data migration and the logic for data selection. The data migration strategy clarifies core elements such as the quantity criteria for migrating data, data filtering rules, and execution pace. It determines which data to select from the source data bucket and how much data to migrate to the target data bucket, serving as the direct basis for the data migration operation.
[0108] In this context, positional relationship can be understood as the hierarchical affiliation and branching relationship between the source and target data buckets within the data decision tree structure. Positional relationship can include structural characteristics such as whether the source and target data buckets belong to the same parent node, are leaf nodes at the same level, or are under the same attribute branch. Positional relationship serves as the core basis for determining the legality of migration and planning migration paths. Migration is only conducted between data buckets at the same level and on the same branch to avoid disrupting the data's attribute classification logic. For example, if both the source and target data buckets belong to intermediate nodes with a "level ≥ 30" and are leaf nodes at the same level, this structural relationship constitutes their positional relationship.
[0109] The data migration path can be understood as the execution link and flow channel for business data to flow from the source data bucket to the target data bucket. The data migration path can be determined by the positional relationship between the source and target data buckets and the data migration strategy, clarifying the outflow nodes, transit nodes, and inflow nodes of the data; it not only ensures that the data migration process follows the structural logic of the decision tree, avoids data attribute classification confusion, but also makes the migration process controllable and traceable.
[0110] In one scenario, upon receiving a data balancing instruction from the system, the instruction is first parsed to extract the first bucket identifier of the source data bucket, the second bucket identifier of the target data bucket, and the corresponding data migration strategy, thus configuring the parameters for this migration task. Further, based on the two bucket identifiers, the nodes and hierarchical branches of the source and target data buckets in the data decision tree are located to confirm their positional relationship. Combined with the migration scale and data selection rules stipulated in the data migration strategy, a compliant and efficient data migration path is determined. Finally, according to the determined data migration path, a portion of business data is selected from the source data bucket according to the migration strategy, migrated, and stored in the target data bucket, completing the data balancing adjustment operation.
[0111] In one scenario, the provided solution, after completing the data migration process for business data within multiple first data buckets and achieving initial load balancing between buckets, in order to further optimize the storage granularity of the data buckets, reduce the index maintenance and traversal scheduling costs caused by a large number of fragmented data buckets, and avoid the long-term dispersed occupation of management resources by buckets with too small a data volume, the system will perform a data volume compliance check on all migrated first data buckets; when the check finds that there are multiple small data buckets with data sizes that do not meet the standards, a bucket merging mechanism will be initiated to consolidate storage resources. The specific execution method may include: in response to the existence of multiple fifth data buckets among multiple first data buckets, performing a merging process on the multiple fifth data buckets.
[0112] The fifth data bucket can be the first data bucket, whose total stored business data volume is below a preset data volume threshold. It is the target of the merging process and belongs to a subset of the first data bucket. The fifth data bucket can also be a small data bucket selected based on data volume during the storage granularity optimization phase after data migration. If the fifth data bucket contains too little data, and the data is scattered, it will increase the index maintenance cost and traversal scheduling overhead of the data decision tree. Therefore, it will be included in the merging scope to optimize the storage structure.
[0113] The preset data volume threshold can be understood as a pre-defined critical value standard used to determine whether the first data bucket is too small in granularity. This threshold can be pre-configured by the system based on indicators such as storage management efficiency, decision tree traversal performance, and resource consumption costs. It serves as the boundary for distinguishing between normal-sized data buckets and the fifth data bucket to be merged. When the total amount of business data in the first data bucket is lower than this threshold, it can be considered that the maintenance cost of a single bucket exceeds the storage benefit, triggering the merge verification logic. The preset data volume threshold can include any value, selectable options such as 40, 50, 80, or 100 records.
[0114] Merging can be understood as a storage structure optimization operation that integrates multiple small fifth data buckets into a unified data bucket. Merging can be performed on multiple fifth data buckets under the same intermediate node and belonging to the same sub-range of attribute dimensions. It aggregates the business data within each bucket into a single data bucket for storage, while simultaneously updating the mounting references of the corresponding leaf nodes in the decision tree. This reduces the total number of data buckets, lowers the overall overhead of index maintenance and traversal scheduling, and improves storage resource utilization. For example, merging two fifth data buckets storing 32 and 28 data entries respectively under the same level branch into a normal-sized data bucket storing 60 data entries is an example of merging.
[0115] In one scenario, after completing the data migration process, a second data volume verification is performed on all first data buckets within the jurisdiction of the same intermediate node. The total amount of business data in each first data bucket is compared with the preset data volume threshold one by one, and all fifth data buckets with insufficient data volume are screened out. When the verification confirms that there are two or more fifth data buckets among multiple first data buckets, the system performs a merging process on the group of fifth data buckets, aggregating all business data in each fifth data bucket into the same data bucket for storage, synchronously updating the data bucket mounting reference relationship of the corresponding leaf node of the decision tree, and canceling the original multiple fifth data buckets, thereby reducing the number of fragmented data buckets and reducing index maintenance overhead.
[0116] The technical solution of this disclosure, by conducting business data volume statistics on the first data bucket associated with each intermediate node in the data decision tree, calculating the corresponding load difference ratio to quantify the degree of load imbalance between buckets, and then performing differentiated processing on the business data according to the degree of load difference, can accurately perceive the data distribution imbalance under each branch, and promptly detect and alleviate the problem of excessively high or low load in local data buckets. It can not only dynamically balance the data carrying capacity of each bucket to avoid high-load buckets becoming performance bottlenecks, but also effectively improve the overall execution efficiency and response stability of decision tree traversal, data filtering and extraction, and make the allocation of storage resources more reasonable, thereby improving the overall resource utilization. At the same time, the load control method based on intermediate node granularity has precise local adjustment capabilities, and can complete load optimization without interfering with the normal operation of other branches, ensuring the efficient and stable operation of the data decision tree throughout its entire lifecycle.
[0117] Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this disclosure. Figure 4 As shown, the data processing device includes: a first module 410, a second module 420, and a third module 430; wherein, the first module is used to receive a data filtering request, the data filtering request including data filtering conditions and data extraction quantity; the second module is used to compile the data filtering conditions to obtain a decision tree traversal instruction, and execute the decision tree traversal instruction on a pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions; wherein, the data decision tree includes a root node, intermediate nodes, and a first data bucket containing stored business data, the root node is used to indicate the highest-level attribute dimension of the business data, the intermediate nodes are used to indicate the second-level attribute dimension of the business data, and the first data bucket contains the second data bucket; the third module is used to determine a third data bucket from at least one second data bucket, and based on the business data stored in the third data bucket and the data extraction quantity, obtain target business data corresponding to the data filtering conditions, and return the target business data as a request response result corresponding to the data filtering request.
[0118] The technical solution of this disclosure, by receiving standardized data filtering requests through a unified entry point, decouples the filtering requirements of the business side from the underlying filtering execution logic. Business parties only need to focus on their own business rules, without needing to be aware of the implementation details of the underlying filtering. The unified request entry point allows for centralized management of all filtering requests, which, compared to the highly coupled implementation of business logic and filtering logic in the full traversal approach, facilitates subsequent iteration and expansion of filtering capabilities. Furthermore, by compiling filtering conditions into decision tree traversal instructions, candidate data buckets are directly located on the tree index structure, eliminating the need to scan all candidate data one by one. This significantly reduces the time overhead of filtering and solves the problem of increased latency with larger data volumes in the full traversal approach. Based on the multi-dimensional index structure of the decision tree, it naturally supports combined filtering of multi-dimensional attributes. Compared to the method of comparing data item by item in full traversal, the filtering efficiency does not significantly decrease with the growth of data scale, meeting the real-time filtering requirements of massive amounts of data. Furthermore, by selecting data buckets first and then directly and randomly reading data within those buckets, the time spent on random extraction can be controlled within a fixed range, not increasing with the total amount of data, thus ensuring filtering performance in high-concurrency scenarios. The extraction method based on direct indexing within buckets eliminates the need for secondary traversal and comparison of data within buckets, significantly reducing the computational overhead of a single extraction compared to the full traversal scheme, and supporting millisecond-level real-time filtering requirements. The technical solution of this disclosure, by constructing a multi-dimensional decision tree to achieve bucket indexing of data, transforms the filtering process into decision tree path positioning and random reading within buckets. This effectively solves the problems of high latency and inability to adapt to massive amounts of data in related technologies using the full traversal scheme, ensuring the performance requirements of real-time filtering. Moreover, the tree structure-based multi-dimensional data indexing can adapt to the combined filtering needs of multiple dimensions and has good scalability.
[0119] In some embodiments of this disclosure, optionally, the apparatus further includes: a fourth module, a fifth module, and a sixth module; wherein, the fourth module is used to acquire a multidimensional data stream and preset data filtering rules, the multidimensional data stream containing multiple business data items, each of the business data items carrying multiple business attribute fields of different dimensions; the fifth module is used to initialize and construct an empty decision tree according to the preset data filtering rules; the empty decision tree includes a root node, intermediate nodes, and a fourth data bucket for writing business data; the sixth module is used to determine the corresponding fourth data bucket for the multiple business data items according to their respective business attribute fields, and write the business data into the corresponding fourth data bucket to obtain a data decision tree.
[0120] In some embodiments of this disclosure, optionally, the second module 420 includes: a first unit, used to process the data filtering conditions using a preset instruction template to obtain a decision tree traversal instruction.
[0121] In some embodiments of this disclosure, optionally, the third module 430 includes: a second unit and a third unit; wherein, the second unit is configured to determine the second data bucket as a third data bucket in response to the number of the second data buckets being one; and the third unit is configured to determine the third data bucket from the multiple second data buckets based on the business data stored in the multiple second data buckets in response to the number of the second data buckets being multiple.
[0122] In some embodiments of this disclosure, optionally, the third unit is specifically used to determine the sampling probability corresponding to the second data bucket based on the amount of business data stored in the second data bucket; to sample the multiple second data buckets based on the sampling probabilities corresponding to the multiple second data buckets respectively, and to determine the sampled second data bucket as the third data bucket.
[0123] In some embodiments of this disclosure, optionally, the third module 430 includes: a fourth unit, a fifth unit, and a sixth unit; wherein, the fourth unit is used to generate a random index using a random generator, taking the total number of business data stored in the third data bucket as the upper limit of the value; the fifth unit is used to directly access the third data bucket according to the random index and read the business data corresponding to the random index; the sixth unit is used to cyclically execute the operations of generating the random index and reading the business data according to the data extraction quantity, and determine the read business data as the target business data corresponding to the data filtering conditions.
[0124] In some embodiments of this disclosure, optionally, the apparatus further includes: a seventh module, configured to add a selection mark to the read business data in response to the data sampling mode being a non-replacement sampling mode.
[0125] In some embodiments of this disclosure, optionally, the apparatus further includes: an eighth module, a ninth module, and a tenth module; wherein the eighth module is configured to obtain the total amount of business data stored in the first data buckets associated with the leaf node in the data decision tree; the ninth module is configured to determine the load difference ratio corresponding to the leaf node based on the total amount corresponding to the multiple first data buckets; and the tenth module is configured to process the business data stored in the multiple first data buckets based on the load difference ratio.
[0126] In some embodiments of this disclosure, optionally, the ninth module includes: a seventh unit and an eighth unit; wherein, the seventh unit is used to determine a minimum and a maximum quantity from the total quantities corresponding to the plurality of first data buckets respectively; the eighth unit is used to determine the ratio between the maximum quantity and the minimum quantity to obtain the load difference ratio corresponding to the leaf node.
[0127] In some embodiments of this disclosure, optionally, the tenth module includes: a ninth unit and a tenth unit; wherein, the ninth unit is used to maintain state detection of the service data stored in the plurality of first data buckets in response to the load difference ratio being less than a ratio threshold; and the tenth unit is used to perform data migration processing on the service data stored in the plurality of first data buckets in response to the load difference ratio not being less than a ratio threshold.
[0128] In some embodiments of this disclosure, optionally, the tenth unit is specifically configured to receive a data balancing instruction, the data balancing instruction including a first bucket identifier of the source data bucket, a second bucket identifier of the target data bucket, and a data migration strategy; determine a data migration path based on the data migration strategy and the positional relationship between the source data bucket and the target data bucket in the data decision tree; and migrate a portion of the business data in the source data bucket to the target data bucket based on the data migration path.
[0129] In some embodiments of this disclosure, optionally, the apparatus further includes: an eleventh module, configured to, after performing data migration processing on the business data stored in the plurality of first data buckets, perform merging processing on the plurality of fifth data buckets in response to the existence of the plurality of fifth data buckets in the plurality of first data buckets, wherein the fifth data bucket is a first data bucket in which the total amount of stored business data is less than a preset data volume threshold.
[0130] The data processing apparatus provided in this disclosure can execute the data processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.
[0131] It is worth noting that the various units and modules included in the above-mentioned data processing device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.
[0132] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0133] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0134] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0135] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.
[0136] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via read-only memory (ROM) 12 and / or communication unit 19. When the computer program is loaded into random access memory (RAM) 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).
[0137] Various implementations of the systems and techniques described above herein can be implemented in digital circuit systems, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.
[0138] Computer programs used to implement the data processing methods of this disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0139] This disclosure provides a computer-readable storage medium storing computer instructions for causing a processor to execute a data processing method, comprising: receiving a data filtering request, the data filtering request including data filtering conditions and a data extraction quantity; compiling the data filtering conditions to obtain decision tree traversal instructions, and executing the decision tree traversal instructions on a pre-constructed data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions; wherein the data decision tree includes a root node, intermediate nodes, and a first data bucket containing stored business data, the root node indicating the highest-level attribute dimension of the business data, the intermediate nodes indicating the second-level attribute dimensions of the business data, and the first data bucket containing the second data bucket; determining a third data bucket from at least one second data bucket, and obtaining target business data corresponding to the data filtering conditions based on the business data stored in the third data bucket and the data extraction quantity, and returning the target business data as a request response result corresponding to the data filtering request.
[0140] In the context of this disclosure, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, also known as flash memory), optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode-ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0143] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.
[0144] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of embodiments of this disclosure.
[0145] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements a data processing method according to any embodiment of this disclosure.
[0146] In implementing a computer program product, computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0147] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.
[0148] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data processing method, characterized in that, include: Receive a data filtering request, the data filtering request including data filtering conditions and the number of data to be extracted; The data filtering conditions are compiled to obtain decision tree traversal instructions, and the decision tree traversal instructions are executed on the pre-built data decision tree to obtain at least one second data bucket corresponding to the data filtering conditions; wherein, the data decision tree includes a root node, intermediate nodes, leaf nodes and a first data bucket mounted on the leaf nodes, the first data bucket stores business data, and the first data bucket includes the second data bucket. A third data bucket is determined from at least one second data bucket, and based on the business data stored in the third data bucket and the number of data extracted, target business data corresponding to the data filtering conditions is obtained, and the target business data is returned as a request response result corresponding to the data filtering request.
2. The data processing method according to claim 1, characterized in that, Also includes: Acquire multidimensional data streams and preset data filtering rules. The multidimensional data streams contain multiple business data items, each of which carries multiple business attribute fields of different levels and dimensions. Based on the preset data filtering rules, an empty decision tree is initialized and constructed; the empty decision tree includes a root node, intermediate nodes, leaf nodes, and a fourth data bucket for the business data to be written, wherein the root node, the intermediate nodes, and the leaf nodes correspond to different levels of attribute dimensions; The business data is assigned to a fourth data bucket based on its respective business attribute fields, and the business data is written into the corresponding fourth data bucket to obtain a data decision tree.
3. The data processing method according to claim 1, characterized in that, The process of compiling the data filtering conditions to obtain decision tree traversal instructions includes: The data filtering conditions are processed using a preset instruction template to obtain decision tree traversal instructions.
4. The data processing method according to claim 1, characterized in that, Determining the third data bucket from at least one of the second data buckets includes: In response to the fact that the number of the second data bucket is one, the second data bucket is determined as the third data bucket; In response to the fact that there are multiple second data buckets, a third data bucket is determined from the multiple second data buckets based on the business data stored in the multiple second data buckets.
5. The data processing method according to claim 4, characterized in that, The step of determining a third data bucket from the multiple second data buckets based on the business data stored within the multiple second data buckets includes: For multiple second data buckets, the sampling probability corresponding to the second data bucket is determined based on the total amount of business data stored in the second data bucket; Based on the sampling probabilities corresponding to the multiple second data buckets, the multiple second data buckets are sampled, and the sampled second data bucket is determined as the third data bucket.
6. The data processing method according to claim 1, characterized in that, The process of obtaining target business data corresponding to the data filtering conditions based on the business data stored in the third data bucket and the number of data extracted includes: The total amount of business data stored in the third data bucket is used as the upper limit of the value, and a random index is generated using a random generator. The third data bucket is accessed directly based on the random index to read the business data corresponding to the random index; The random index generation and business data reading operations are performed cyclically according to the data extraction quantity, and the read business data is determined as the target business data corresponding to the data filtering conditions.
7. The data processing method according to claim 1, characterized in that, Also includes: For the multiple first data buckets associated with the intermediate node in the data decision tree, obtain the total amount of business data stored in the first data bucket; Based on the total number of the multiple first data buckets, determine the load difference ratio corresponding to the intermediate node; The business data stored in multiple first data buckets is processed based on the load difference ratio.
8. The data processing method according to claim 7, characterized in that, The processing of business data stored in multiple first data buckets based on the load difference ratio includes: In response to the load difference ratio being less than the ratio threshold, the status detection of the business data stored in the multiple first data buckets is maintained. In response to the load difference ratio not being less than the ratio threshold, data migration processing is performed on the business data stored in the multiple first data buckets.
9. The data processing method according to claim 8, characterized in that, The data migration process performed on the business data stored in the multiple first data buckets includes: Receive a data balancing instruction, which includes the first bucket identifier of the source data bucket, the second bucket identifier of the target data bucket, and a data migration strategy; Based on the data migration strategy and the positional relationship between the source data bucket and the target data bucket in the data decision tree, the data migration path is determined; Based on the data migration path, some business data in the source data bucket is migrated to the target data bucket.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-9.
