Method, system and equipment for dynamically classifying and aggregating multi-dimensional data and medium

Abstract

The invention is applicable to the technical field of computers, and provides a method for dynamically classifying and aggregating multi-dimensional data, which comprises the following steps of: acquiring a multi-dimensional data table, analyzing a dimension structure of the multi-dimensional data table, extracting a key field for each dimension, determining a field type and a grouping priority through a preset mapping table, and generating a metadata framework; traversing the metadata framework by adopting a recursive algorithm, determining dimension nodes of the multi-dimensional data table, and performing fragmentation processing on the multi-dimensional data table based on the dimension nodes to obtain a grouping framework; and packaging the grouping framework by adopting a functional programming method to obtain an aggregation result, so that the efficiency and the flexibility of multi-dimensional data analysis are remarkably improved while the data processing accuracy is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, system, device, and medium for dynamic classification and aggregation of multidimensional data. Background Art

[0002] In data processing and analysis scenarios in the software development field, it is usually necessary to group and count complex multidimensional data to improve the efficient use of system resources. However, with the continuous expansion of data scale and the increasing complexity of business needs, traditional data processing methods have gradually exposed their limitations.

[0003] Existing multidimensional data processing techniques often rely on nested mapping structures when grouping multidimensional data. While this approach works, it rapidly increases code complexity when faced with dynamically changing dimensional requirements. Each new grouping dimension requires manual expansion of the nesting level, resulting in reduced code readability and increased maintenance costs. Furthermore, the code lacks flexibility and cannot quickly adapt to changes in business logic, forcing developers to frequently refactor code, leading to low efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a method, system, device and medium for dynamic classification and aggregation of multidimensional data, which can solve one of the above-mentioned problems in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for dynamically classifying and aggregating multidimensional data, comprising:

[0006] Obtain a multidimensional data table, parse the dimensional structure of the multidimensional data table, extract key fields for each dimension, determine the field type and grouping priority through a preset mapping table, and generate a metadata framework;

[0007] Using a recursive algorithm to traverse the metadata framework, determine the dimension nodes of the multidimensional data table, and based on the dimension nodes, perform sharding processing on the multidimensional data table to obtain a grouping framework;

[0008] Functional programming is used to encapsulate the grouping framework to obtain aggregation results.

[0009] Furthermore, the step of obtaining a multidimensional data table, parsing the dimensional structure of the multidimensional data table, extracting key fields for each dimension, determining field types and grouping priorities through a preset mapping table, and generating a metadata framework includes:

[0010] Parsing the hierarchical relationship of each field in the multidimensional data table to generate a tree structure;

[0011] Extracting the field name and parent node path of each leaf node in the tree structure respectively to generate a field set list;

[0012] Match the field set list with the preset mapping table to determine the field type of each field;

[0013] Determining the grouping priority of each field according to the order and frequency of occurrence of the field names in the multidimensional data table;

[0014] Based on the grouping priority, the field name, field type and tree structure of each leaf node are encapsulated to generate a metadata framework.

[0015] Furthermore, the sharding process is performed on the multidimensional data table based on the dimension nodes to obtain a grouping framework, including:

[0016] According to business needs, a dimension node is selected, and the data in the multidimensional data table is fragmented using a hash modulo method to obtain a dimension subset;

[0017] According to business rules, corresponding processing logic is mapped to the dimension subset to generate an intermediate structure list, and cluster analysis is performed on the intermediate structure list to generate a grouping framework, where the grouping framework includes multiple grouping dimensions.

[0018] Furthermore, after obtaining the grouping framework, the method further includes: using a hash table to store the grouping key-value pairs, and performing secondary sharding processing to generate a sharding framework, specifically including:

[0019] For each of the grouping dimensions, a hash table is used to store grouping key-value pairs, where the key represents the grouping field and the value represents the data slice, and the data slice is used to store the multidimensional metadata corresponding to the grouping field;

[0020] Each data shard is marked with a shard number, and a mapping relationship between a key and a data shard is established.

[0021] Furthermore, the secondary sharding process further includes:

[0022] Based on the shard load balancing threshold, merge low-load shards or split high-load shards to generate a storage sharding plan;

[0023] Allocate a computing node to each shard in the storage sharding scheme.

[0024] Furthermore, the method of encapsulating the grouping framework using a functional programming method to obtain an aggregation result includes:

[0025] Defining a data structure of a grouping dimension in the grouping framework, and constructing a grouping function, wherein the grouping function is used to group the multidimensional data table based on the grouping framework;

[0026] By traversing each grouping dimension, outputting the statistical information of each group and the dimension grouping results, an aggregation result is generated.

[0027] Furthermore, before building the grouping function, include:

[0028] Extracting the variable declaration of each of the grouping dimensions as a dimension set, and generating a type constraint rule based on the dimension set;

[0029] Compare the type constraint rules with the variable types of each grouping dimension, and if a type inconsistency is detected, record the incorrect variable type;

[0030] If the error frequency exceeds a preset threshold, the grouping dimension is marked as being in an unsafe state, and the grouping dimension is corrected.

[0031] In a second aspect, an embodiment of the present application provides a multi-dimensional data dynamic classification and aggregation system, including:

[0032] The first processing module is used to obtain a multidimensional data table, analyze the dimensional structure of the multidimensional data table, extract key fields for each dimension, determine the field type and grouping priority through a preset mapping table, and generate a metadata framework;

[0033] A second processing module is configured to traverse the metadata framework using a recursive algorithm to determine dimension nodes of the multidimensional data table, and perform sharding processing on the multidimensional data table based on the dimension nodes to obtain a grouping framework;

[0034] The third processing module is used to encapsulate the grouping framework using a functional programming method to obtain an aggregation result.

[0035] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for dynamic classification and aggregation of multidimensional data when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned method for dynamic classification and aggregation of multidimensional data when executed by a processor.

[0037] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0038] The present application discloses a method for dynamic classification and aggregation of multidimensional data. By parsing the dimensional structure of a multidimensional data table and extracting key fields, the method can deeply understand the inherent logic and hierarchical relationships of the data. At the same time, the method uses a preset mapping table to determine the field type and grouping priority. The generated metadata framework provides a clear and accurate definition and structured representation for the data, making data management more standardized and efficient, and facilitating the accuracy and consistency of subsequent data processing and analysis. In addition, a recursive algorithm is used to traverse the metadata framework to determine the dimension nodes, and the multidimensional data table is sliced based on the dimension nodes to obtain a grouping framework. This method can flexibly divide the data into different logical units according to the actual structure and characteristics of the data, providing a solid foundation for subsequent grouping processing. At the same time, the slicing processing helps to improve the parallelism and scalability of data processing, especially when processing large-scale data, which can significantly improve processing efficiency. In addition, the grouping framework is encapsulated using a functional programming method to obtain the aggregation result, making the code more concise, clear, easy to understand and maintain. At the same time, the modular nature of functional programming makes the implementation of aggregation operations more flexible and reusable, and can easily perform aggregation processing on different grouping frameworks to meet diverse business needs, thereby significantly improving the efficiency and flexibility of multidimensional data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 This is a flow chart of a method for dynamic classification and aggregation of multidimensional data provided by one embodiment of the present invention;

[0041] Figure 2 This is a structural diagram of a multi-dimensional data dynamic classification and aggregation system provided by one embodiment of the present invention;

[0042] Figure 3 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0044] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0045] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0046] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0047] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0048] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0049] See also Figure 1 As shown, the present invention is a multi-dimensional data dynamic classification and aggregation method, comprising the following steps:

[0050] S100: Obtain a multidimensional data table, parse the dimensional structure of the multidimensional data table, extract key fields for each dimension, determine the field type and grouping priority through a preset mapping table, and generate a metadata framework;

[0051] In this embodiment, by parsing the dimensional structure of the multidimensional data table and extracting key fields, we can deeply understand the internal logic and hierarchical relationship of the data. At the same time, with the help of a preset mapping table, we can determine the field type and grouping priority. The generated metadata framework provides a clear and accurate definition and structured representation for the data, making data management more standardized and efficient, and contributing to the accuracy and consistency of subsequent data processing and analysis.

[0052] In some embodiments, step S100 includes:

[0053] Parsing the hierarchical relationship of each field in the multidimensional data table to generate a tree structure;

[0054] Extracting the field name and parent node path of each leaf node in the tree structure respectively to generate a field set list;

[0055] Match the field set list with the preset mapping table to determine the field type of each field;

[0056] Determining the grouping priority of each field according to the frequency of occurrence of the field name in the multidimensional data table;

[0057] Based on the grouping priority, the field name, field type and tree structure of each leaf node are encapsulated to generate a metadata framework.

[0058] In this embodiment, a multidimensional data table to be classified and aggregated by the user is obtained, and by parsing the hierarchical relationship between different dimensions, the dimension division result is output as a tree structure. Specifically, a multidimensional data table contains multiple fields, and a field may contain one or more hierarchical sub-fields, ultimately forming a tree-like hierarchical structure. For example, a sales data table containing multiple dimensions such as products, sales, customers, time, and regions, the hierarchical relationship of its time field may include year → quarter → month → day, and the hierarchical relationship of its region field may be country → province → city → region. In one possible embodiment, SQL window functions are used to parse the hierarchical relationship between different dimensions, such as using statements such as CONNECTBY or WITHRECURSIVE to construct a hierarchical query, thereby obtaining a hierarchical structure of different fields, and ultimately forming a tree structure.

[0059] In this embodiment, the field names and parent node paths of the leaf nodes are extracted from the tree structure to generate a field set list, which is then matched with a preset mapping table. Specifically, the preset mapping table contains three types of field definitions: string type, numeric type, and time type. At the same time, the OpenRefine tool is used to automatically classify unmatched fields. In addition, the priority score is calculated based on the frequency of occurrence of the field in the multidimensional data table to generate a sorted field priority list. Then, the field name, field type, and parent node path are written into the JSON template in the order of the priority list to construct a metadata framework, such as {"name":"sales amount","

[0060] {"type":"numeric","path":"sales->amount"}.

[0061] In some embodiments, if a field does not match the mapping table, the field value is clustered with existing types by calculating cosine similarity, the new type is added to the mapping table, and the matching rules are recorded. Specifically, if a new field "Customer Rating" is found in a multidimensional data table, the cosine similarity method is used to calculate similarity between it and the existing fields in the multidimensional data table, and then it is classified as a numeric type. This new type is added to the mapping table, and the matching rules are simultaneously recorded with the preset mapping table, thereby continuously improving and updating the mapping table and improving the accuracy and efficiency of data processing.

[0062] Specifically, taking a multidimensional data table of VIP vouchers in a game as an example, the multidimensional data table is as follows:

[0063] “{ID:1,Name:"VIP1-Promotion Ticket",Level:1,Type:1,DeductionAmount:7.0},

[0064] {ID:2,Name:"VIP1-Weekly Ticket",Level:1,Type:2,DeductionAmount:5.0},

[0065] {ID:2,Name:"VIP1-Weekly Ticket",Level:1,Type:2,DeductionAmount:5.0},

[0066] {ID:2,Name:"VIP1-Weekly Ticket",Level:1,Type:2,DeductionAmount:5.0},

[0067] {ID:3,Name:"VIP2-Promotion Ticket",Level:2,Type:1,DeductionAmount:9.0},

[0068] {ID:4,Name:"VIP2-Weekly Ticket",Level:2,Type:2,DeductionAmount:7.0},

[0069] {ID:4,Name:"VIP2-Weekly Ticket",Level:2,Type:2,DeductionAmount:7.0},

[0070] {ID:4,Name:"VIP2-Weekly Ticket",Level:2,Type:2,DeductionAmount:7.0},

[0071] ...”

[0072] By parsing the above multidimensional data table, we determine that it contains five fields: ID, Name, Level, Type, and DeductionAmount. For each field, we analyze its corresponding hierarchical relationship. Among them, ID, Level, Type, and DeductionAmount do not have a hierarchical relationship, and the corresponding tree structure is a single hierarchical structure. For the Name field, combined with the Level and Type fields, we determine that it has the following hierarchical relationship: VIP level -> voucher type, that is, Level -> Type. Based on this, we construct a tree structure. Based on this, the generated field set list is as follows:

[0073] {"ID":"1","Path":""},{"ID":"2","Path":""}...

[0074] {"Name":"VIP1-Promotion Ticket","Path":"Level 1->Type 1"},

[0075] {"Name":"VIP1-Weekly Ticket","Path":"Level 1->Type 2"},

[0076] {"Name":"VIP2-Promotion Ticket","Path":"Level 2->Type 1"},

[0077] {"Name":"VIP2-Weekly Ticket","Path":"Level 2->Type 2"}...

[0078] {"Level":"1","Path":""},{"Level":"2","Path":""}...

[0079] {"Type":"1","Path":""},{"Type":"2","Path":""}...

[0080] {"DeductionAmount":"5.0","Path":""},

[0081] {"DeductionAmount":"7.0","Path":""}...

[0082] Match each field in the above field set list with the preset mapping table to determine the field type of each field. In addition, since ID, Name, Level, Type, and DeductionAmount appear at the same frequency in the multidimensional data table, grouping and prioritization are performed according to the order in which each field appears in the multidimensional data table. Based on this, a metadata framework is constructed, as shown below:

[0083] {"name":"ID","type":"numeric","path":"ID"},

[0084] {"name":"Name","type":"String","path":"Name->Level->Type"},

[0085] {"name":"Level","type":"Numeric","path":"Level"},

[0086] {"name":"Type","type":"Numeric","path":"Type"},

[0087] {"name":"DeductionAmount","type":"Number","path":"DeductionAmount"}

[0088] It is understandable that when there is too much multidimensional data, the data is messy and disordered. Therefore, this application parses and reconstructs the multidimensional data through the above step S100, which can clearly understand the organization of each data and the relationship between each dimension, thereby providing a data basis for the subsequent classification and aggregation of multidimensional data.

[0089] S200: traverse the metadata framework using a recursive algorithm to determine dimension nodes of the multidimensional data table, and perform sharding processing on the multidimensional data table based on the dimension nodes to obtain a grouping framework;

[0090] In this embodiment, a recursive algorithm is used to traverse the metadata framework. Specifically, it traverses the fields within the metadata framework to determine the dimension nodes within the multidimensional data table. The recursive algorithm allows for dynamic identification of fields within the metadata framework based on its structure, where fields correspond to dimension nodes. Therefore, if the data structure within the multidimensional data table changes, the corresponding dimension nodes can be retrieved through the recursive algorithm. Furthermore, sharding based on dimension nodes provides greater flexibility, adapting to business expansion and improving data processing efficiency.

[0091] In some embodiments, step S200 includes:

[0092] According to business needs, a dimension node is selected, and the data in the multidimensional data table is fragmented using a hash modulo method to obtain a dimension subset;

[0093] According to business rules, corresponding processing logic is mapped to the dimension subset to generate an intermediate structure list, and cluster analysis is performed on the intermediate structure list to generate a grouping framework, where the grouping framework includes multiple grouping dimensions.

[0094] In this embodiment, the data in the multidimensional data table can be divided into different dimension subsets according to different dimension nodes. For example, in the multidimensional data table of the VIP voucher mentioned above, it can be divided into two dimension subsets according to the dimension node ID, or divided into four dimension subsets according to the dimension node Name. Based on this, according to actual business needs, a dimension node is selected to shard the multidimensional data table to obtain dimension subsets. Specifically, if the actual business needs of the multidimensional data table of the VIP voucher mentioned above are to classify and display the VIP level, voucher type (promotion voucher or weekly voucher, etc.) and number of vouchers with the same discount of the VIP voucher, then the dimension node Level is selected to shard the multidimensional data table. Through sharding, a large number of dimension nodes are dispersed into multiple dimension subsets, so that the data volume of each dimension subset is relatively small, which is convenient for subsequent parallel processing and improves processing efficiency. At the same time, sharding also helps to distribute workload among multiple processing units to achieve load balancing.

[0095] In this embodiment, the obtained dimension subsets are logically processed according to the business rules corresponding to their actual business needs, thereby generating an intermediate structure list. It can be understood that the intermediate structure list refers to the intermediate results obtained after a series of transformations, calculations, or aggregations during the data processing or analysis process. These intermediate results may exist in the form of lists, tables, or graphs, and contain key features of the data or intermediate states during the processing process. By organizing and managing them, the subsequent classification and aggregation calculation process is simplified. For example, for the multiple dimension subsets obtained after the above-mentioned dimensional node Level sharding, according to their business rules, a voucher type table and a voucher table with the same discount can be separately counted for each dimension subset, such as the Level 1 promotion voucher statistics table and the Level 2 weekly voucher statistics table. This can be used to analyze information such as the VIP level, voucher type (promotion voucher or weekly voucher, etc.), and the number of vouchers with the same discount.

[0096] In this embodiment, cluster analysis is performed on the multiple generated intermediate structure lists to generate a grouping framework. For example, the intermediate structure lists related to the voucher type in the above-mentioned dimension subset Level 1 can be clustered to generate a voucher type table, or the intermediate structure lists related to the discount information in the dimension subset Level 1 can be clustered to generate a discount information statistics table. Based on this, the multidimensional data table is mapped into multiple grouping dimensions, such as the voucher type dimension at Level 1 and the discount information dimension at Level 1.

[0097] In some embodiments, after obtaining the grouping framework, the method further includes: using a hash table to store the grouping key-value pairs, and performing secondary sharding to generate a sharding framework, specifically including:

[0098] For each of the grouping dimensions, a hash table is used to store grouping key-value pairs, where the key represents the grouping field and the value represents the data slice, and the data slice is used to store the multidimensional metadata corresponding to the grouping field;

[0099] Each data shard is marked with a shard number, and a mapping relationship between a key and a data shard is established.

[0100] In this embodiment, a hash table structure, such as HashMap, is used to store key-value pairs, where the key is the grouping field and the value is the data shard, thereby forming a sharding framework based on the grouping key. For example, in the aforementioned Level 1 voucher type dimension, each voucher type represents a grouping field, and the multidimensional metadata set corresponding to the voucher type is the data shard. Specifically, in one possible embodiment, in the aforementioned Level 1 voucher type dimension, when the voucher type is a promotion voucher, there are 100 corresponding multidimensional metadata items. In this way, the "promotion voucher" key is mapped to a data shard containing 100 records of promotion vouchers.

[0101] In this embodiment, according to the key-value pair data, a modulo operation is performed based on the hash value of the key to determine the data shard in which each key-value pair is stored, and the shard number is marked for the data shard, thereby establishing a mapping relationship between the key and the data shard position, which is convenient for subsequent data expansion.

[0102] In some embodiments, the performing secondary sharding further includes:

[0103] Based on the shard load balancing threshold, merge low-load shards or split high-load shards to generate a storage sharding plan;

[0104] Allocate a computing node to each shard in the storage sharding scheme.

[0105] In this embodiment, for the sharding framework, low-load shards are merged or high-load shards are split according to the load balancing threshold of the shards to generate a storage sharding scheme. Specifically, in a possible embodiment, if the sharding load balancing threshold is to store 50 records for each sharding framework, and one of the sharding frameworks has only 20 records, then it is merged with the 30 records in the other sharding framework into a new sharding framework, and the corresponding sharding number is marked on it; if there is a sharding framework with 80 records, then it is split into two sharding frameworks of 40 each. The above sharding adjustment ensures a more reasonable allocation of computing resources and improves the efficiency of classification aggregation.

[0106] In some embodiments, if the data volume of a single sharding framework exceeds a preset sharding load balancing threshold, the Spark executor memory parameters are increased, and the reduceByKey operation is re-executed to merge the values of the same key and generate an intermediate result set. The key distribution density of each shard is obtained from the intermediate result set. When the density variance exceeds the preset chi-square test threshold, the chain address method is used to reconstruct the hash index of the HashMap to reduce the hash conflict rate and output an efficient grouping result. The key query time based on the efficient grouping result is compared with the query performance of the initial grouping structure. If the improvement is less than 20%, a Bloom filter is used to pre-filter the key value and finally generate the grouping output. The key distribution uniformity and query latency indicators are extracted from the final grouping output, and the Gini coefficient and percentile are calculated using Pandas to verify the statistical significance of the key distribution under type safety constraints. Based on this, the efficiency of data classification and aggregation in multidimensional data tables is improved.

[0107] S300: Encapsulate the grouping framework using a functional programming method to obtain an aggregation result.

[0108] In this embodiment, functional programming is used to encapsulate the grouping framework to obtain aggregation results, making the code more concise, clear, easy to understand and maintain. At the same time, the modular characteristics of functional programming make the implementation of aggregation operations more flexible and reusable, and can easily perform aggregation processing on different grouping frameworks to meet diverse business needs, thereby significantly improving the efficiency and flexibility of multidimensional data analysis.

[0109] In some embodiments, step S300 includes:

[0110] Defining a data structure of a grouping dimension in the grouping framework, and constructing a grouping function, wherein the grouping function is used to group the multidimensional data table based on the grouping framework;

[0111] By traversing each grouping dimension, outputting the statistical information of each group and the dimension grouping results, an aggregation result is generated.

[0112] In this embodiment, by defining the data structure of the grouping dimension in the grouping framework and constructing a grouping function, it is possible to accurately and efficiently perform grouping operations on multidimensional data tables based on clear and standardized grouping logic, avoiding confusion or errors in the grouping process, and greatly improving the accuracy and efficiency of data processing. On the basis of grouping, by traversing each grouping dimension and outputting the statistical information and dimensional grouping results of each group, it is possible to fully and carefully understand the data characteristics and distribution within each group, providing rich and valuable information for subsequent data analysis. Based on the above operations, aggregation results are generated, which can quickly integrate scattered multidimensional data into aggregated data with higher-level information value.

[0113] In some embodiments, before constructing the grouping function, the following steps are included:

[0114] Extracting the variable declaration of each of the grouping dimensions as a dimension set, and generating a type constraint rule based on the dimension set;

[0115] Compare the type constraint rules with the variable types of each grouping dimension, and if a type inconsistency is detected, record the incorrect variable type;

[0116] If the error frequency exceeds a preset threshold, the grouping dimension is marked as being in an unsafe state, and the grouping dimension is corrected.

[0117] In this embodiment, the variable declaration of each grouping dimension is parsed. For example, the variable of the voucher type dimension of Level 1 is a voucher type. By parsing the variable definition, the variable type corresponding to the variable can be clarified. For example, the variable type corresponding to the voucher type is a mapping hierarchical structure. Based on this, the static code analysis tool Checkstyle is used to generate type constraint rules. For example, the "voucher type" is set to a predefined hierarchical enumeration value that must comply with the preset predefined hierarchy, such as 1 corresponding to the promotion voucher and 2 corresponding to the weekly voucher. Or the "discount information" is set to a numeric type. If a string is entered, it is considered a violation. It can be understood that the type constraint rules are set based on actual business to meet business needs.

[0118] In this embodiment, the actual variable type of each grouping dimension is matched with the type constraint rule. For example, the AST node comparison method is used to check whether the type of each variable or expression conforms to the type constraint rule by traversing the AST. If type inconsistency is detected, it is marked as an error type. When the error frequency exceeds the preset threshold, it means that the variable type setting of the current grouping dimension is unreasonable, making it in an unsafe state. Therefore, the grouping dimension needs to be corrected, such as inputting the grouping dimension in an unsafe state into the rule engine, and then readjusting the rules according to the variable usage scenario. It can be understood that by correcting the variables of the grouping dimension, the stability and security of the data can be guaranteed when new multidimensional data is input or new dimension nodes are introduced.

[0119] See also Figure 2 As shown, the present invention also provides a multi-dimensional data dynamic classification and aggregation system, the system comprising:

[0120] The first processing module 201 is used to obtain a multidimensional data table, analyze the dimensional structure of the multidimensional data table, extract key fields for each dimension, determine the field type and grouping priority through a preset mapping table, and generate a metadata framework;

[0121] The second processing module 202 is configured to traverse the metadata framework using a recursive algorithm to determine dimension nodes of the multidimensional data table, and perform sharding processing on the multidimensional data table based on the dimension nodes to obtain a grouping framework;

[0122] The third processing module 203 is used to encapsulate the grouping framework using a functional programming method to obtain an aggregation result.

[0123] It is understandable that if Figure 1 The contents of the embodiment of the multidimensional data dynamic classification and aggregation method shown in the figure are applicable to the embodiment of the multidimensional data dynamic classification and aggregation system. The functions specifically implemented by the embodiment of the multidimensional data dynamic classification and aggregation system are similar to those in the embodiment of the figure. Figure 1 The embodiment of the multidimensional data dynamic classification and aggregation method shown is the same as that of the embodiment of the multidimensional data dynamic classification and aggregation method shown in FIG. Figure 1 The beneficial effects achieved by the embodiment of the multidimensional data dynamic classification and aggregation method shown are also the same.

[0124] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] See also Figure 3 As shown, an embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, a multidimensional data dynamic classification and aggregation method as described in any one of the above methods is implemented.

[0127] The computer device 3 may be a desktop computer, a notebook computer, a PDA, a cloud server or other computing devices. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that Figure 3 This is merely an example of the computer device 3 and does not constitute a limitation on the computer device 3 . The computer device 3 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 3 may also include input and output devices, network access devices, etc.

[0128] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0129] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 3. Furthermore, the memory 302 may include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 302 may also be used to temporarily store data that has been output or is about to be output.

[0130] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for dynamic classification and aggregation of multidimensional data as described in any one of the above methods is implemented.

[0131] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to a camera / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard drive, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0132] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for dynamic classification and aggregation of multidimensional data, characterized in that: include: Obtain a multidimensional data table, parse the dimensional structure of the multidimensional data table, extract key fields for each dimension, determine the field type and grouping priority through a preset mapping table, and generate a metadata framework; Using a recursive algorithm to traverse the metadata framework, determine the dimension nodes of the multidimensional data table, and based on the dimension nodes, perform sharding processing on the multidimensional data table to obtain a grouping framework; Functional programming is used to encapsulate the grouping framework to obtain aggregation results.

2. The method according to claim 1, wherein The method of obtaining a multidimensional data table, parsing the dimensional structure of the multidimensional data table, extracting key fields for each dimension, determining field types and grouping priorities through a preset mapping table, and generating a metadata framework includes: Parsing the hierarchical relationship of each field in the multidimensional data table to generate a tree structure; Extracting the field name and parent node path of each leaf node in the tree structure respectively to generate a field set list; Match the field set list with the preset mapping table to determine the field type of each field; Determining the grouping priority of each field according to the order and frequency of occurrence of the field names in the multidimensional data table; Based on the grouping priority, the field name, field type and tree structure of each leaf node are encapsulated to generate a metadata framework.

3. The method according to claim 1, wherein The slicing process of the multidimensional data table based on the dimension nodes to obtain a grouping framework includes: According to business needs, a dimension node is selected, and the data in the multidimensional data table is fragmented using a hash modulo method to obtain a dimension subset; According to business rules, corresponding processing logic is mapped to the dimension subset to generate an intermediate structure list, and cluster analysis is performed on the intermediate structure list to generate a grouping framework, where the grouping framework includes multiple grouping dimensions.

4. The method according to claim 3, wherein After obtaining the grouping framework, the method further includes: using a hash table to store the grouping key-value pairs, and performing secondary sharding processing to generate a sharding framework, specifically including: For each of the grouping dimensions, a hash table is used to store grouping key-value pairs, where the key represents the grouping field and the value represents the data slice, and the data slice is used to store the multidimensional metadata corresponding to the grouping field; Each data shard is marked with a shard number, and a mapping relationship between a key and a data shard is established.

5. The method according to claim 4, wherein The secondary sharding process further includes: Based on the shard load balancing threshold, merge low-load shards or split high-load shards to generate a storage sharding plan; Allocate a computing node to each shard in the storage sharding scheme.

6. The method according to claim 1, wherein The method of encapsulating the grouping framework using a functional programming method to obtain an aggregation result includes: Defining a data structure of a grouping dimension in the grouping framework, and constructing a grouping function, wherein the grouping function is used to group the multidimensional data table based on the grouping framework; By traversing each grouping dimension, outputting the statistical information of each group and the dimension grouping results, an aggregation result is generated.

7. The method according to claim 1, wherein Before building your grouping function, include: Extracting the variable declaration of each of the grouping dimensions as a dimension set, and generating a type constraint rule based on the dimension set; Compare the type constraint rules with the variable types of each grouping dimension, and if a type inconsistency is detected, record the incorrect variable type; If the error frequency exceeds a preset threshold, the grouping dimension is marked as being in an unsafe state, and the grouping dimension is corrected.

8. A multi-dimensional data dynamic classification and aggregation system, characterized by: include: The first processing module is used to obtain a multidimensional data table, analyze the dimensional structure of the multidimensional data table, extract key fields for each dimension, determine the field type and grouping priority through a preset mapping table, and generate a metadata framework; A second processing module is configured to traverse the metadata framework using a recursive algorithm to determine dimension nodes of the multidimensional data table, and perform sharding processing on the multidimensional data table based on the dimension nodes to obtain a grouping framework; The third processing module is used to encapsulate the grouping framework using a functional programming method to obtain an aggregation result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images