Data aggregation processing method and device, computer equipment and storage medium

Abstract

The invention discloses a data aggregation processing method and device, computer equipment and a storage medium, belongs to the technical field of big data, and is applied to data aggregation in the financial field. The method comprises the following steps: firstly, grouping dimension fields by utilizing a mapping rule and a hierarchical clustering algorithm; and secondly, dynamic grouping mapping is established, one-time scanning is performed on the source data table, and each data record is directly attributed to a corresponding grouping structure, so that the data processing efficiency is improved. Meanwhile, a correlation index of the index and the data partition is constructed based on a Hash mapping algorithm. Under the support of an indexed data structure, the aggregation calculation function can efficiently execute operations such as summation and averaging, and rapid aggregation of multi-dimensional data is realized. Generally speaking, according to the method, the correctness of an aggregation result is ensured, meanwhile, the calculation and storage pressure of the system is remarkably reduced, the timeliness and expandability of data analysis are improved, and the method is particularly suitable for real-time or quasi-real-time multi-dimensional analysis requirements in a large-scale business scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of big data, and particularly relates to a data aggregation processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] In the field of big data processing, multi-dimensional aggregation technology is the core support for data analysis and decision-making, and is widely used in business intelligence, data warehouse and decision support system scenarios. By grouping data according to multiple dimensions (such as time, region, category) and calculating aggregated values such as sum, count or average, enterprises can extract multi-angle insights from massive data to support accurate decision-making. The importance of this technology lies in its ability to transform complex business data into actionable analysis results. For example, e-commerce enterprises analyze user region, purchase time period and commodity category combination data to optimize promotion strategies; logistics companies aggregate transportation methods and regional data to reduce delivery costs. These application scenarios show that multi-dimensional aggregation is not only the basis of data processing, but also a key technology for enterprises to improve efficiency and competitiveness.

[0003] However, existing methods have significant defects when dealing with complex business scenarios. Traditional multi-dimensional aggregation usually relies on directly applying fixed dimension combination formulas, and performs independent aggregation operations for different analysis requirements. This approach is inefficient and resource-intensive when faced with dynamic or diverse business requirements. In particular, when the business needs to analyze the same data set with multiple dimension combinations, the data table often needs to be read multiple times, resulting in repeated occupation of system resources such as disk throughput and network bandwidth. This repeated operation not only increases processing time, but also puts a heavy burden on system performance. For example, if an e-commerce enterprise needs to analyze purchase time period data divided by 24 hours and by morning, afternoon, evening and early morning, the traditional method will read the source data table separately and repeatedly perform aggregation operations, significantly increasing resource consumption.

[0004] Therefore, how to efficiently implement dynamic dimension grouping in multi-dimensional aggregation while adapting to complex business requirements without repeatedly reading source data and reducing resource consumption has become a key problem in the field of data processing that needs to be solved. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a data aggregation processing method and device, computer equipment and a storage medium to efficiently implement dynamic dimension grouping in multi-dimensional aggregation while adapting to complex business requirements without repeatedly reading source data and reducing resource consumption.

[0006] To solve the above technical problems, the embodiments of the present application provide a data aggregation processing method, which adopts the following technical solutions:

[0007] A method for data aggregation processing, comprising:

[0008] A dimension field is extracted from a source data table, a preset mapping rule is used to preliminarily classify the dimension field, and an initial dimension set is obtained;

[0009] A hierarchical clustering algorithm is applied to generate a dynamic grouping structure according to the field attributes in the initial dimension set, and if the dynamic grouping structure matches a preset standard structure, grouping mapping is performed on the dynamic grouping structure, and a dynamic grouping mapping is obtained;

[0010] Each data record scanned is allocated to the corresponding grouping structure according to the dynamic grouping mapping, and a data partition is obtained;

[0011] An index association between an index and the data partition is constructed by using a hash mapping algorithm, and it is judged whether the index association covers all dimension combinations, and if all are covered, an indexed data structure is obtained;

[0012] According to the indexed data structure, a summation or an average value is calculated for each dimension combination by executing an aggregation calculation function, and an aggregation result set is obtained.

[0013] To solve the above technical problems, the embodiment of the application further provides a data aggregation processing device, which adopts the technical scheme as follows:

[0014] A data aggregation processing device, comprising:

[0015] A dimension classification module is configured to extract a dimension field from a source data table, use a preset mapping rule to preliminarily classify the dimension field, and obtain an initial dimension set;

[0016] A grouping mapping module is configured to apply a hierarchical clustering algorithm to generate a dynamic grouping structure according to the field attributes in the initial dimension set, and if the dynamic grouping structure matches a preset standard structure, perform grouping mapping on the dynamic grouping structure, and obtain a dynamic grouping mapping;

[0017] A data table scanning module is configured to perform a one-time scan on the source data table according to the dynamic grouping mapping, allocate each data record scanned to the corresponding grouping structure, and obtain a data partition;

[0018] An association index module is configured to extract an index value sequence from the data partition, construct an index association between an index and the data partition by using a hash mapping algorithm, and judge whether the index association covers all dimension combinations, and if all are covered, obtain an indexed data structure;

[0019] An aggregation operation module is configured to calculate a summation or an average value for each dimension combination by executing an aggregation calculation function according to the indexed data structure, and obtain an aggregation result set.

[0020] To solve the above technical problems, the embodiment of the present application also provides a computer device which adopts the technical scheme as follows:

[0021] A computer device comprises a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the method for data aggregation processing according to any one of the above.

[0022] To solve the above technical problems, the embodiment of the present application also provides a computer readable storage medium which adopts the technical scheme as follows:

[0023] A computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to realize the steps of the method for data aggregation processing according to any one of the above.

[0024] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0025] The present application discloses a method and device for data aggregation processing, a computer device and a storage medium, belonging to the field of big data technology and applied to data aggregation in the financial field. Firstly, the dimension fields are grouped by using a mapping rule and a hierarchical clustering algorithm, which can flexibly adapt to analysis requirements of different granularities while ensuring consistency of business logic, avoiding data sparseness or deviation caused by unreasonable dimension splitting. Secondly, by establishing a dynamic grouping mapping and performing a one-time scan on the source data table, each data record is directly attributed to the corresponding grouping structure, reducing the overhead of multiple accesses to the source table and greatly improving the data processing efficiency. At the same time, based on the hash mapping algorithm, an associated index of indicators and data partitions is constructed, which can not only quickly locate the indicator value under any dimension combination, but also ensure full coverage of the index to the dimension combination, thereby improving the completeness and accuracy of the results. With the support of the indexed data structure, the aggregation calculation function can efficiently perform summation, average value and other operations, realizing fast aggregation of multi-dimensional data. Overall, while ensuring the correctness of the aggregation results, the present application significantly reduces the computing and storage pressure of the system, improves the timeliness and scalability of data analysis, and is particularly suitable for real-time or quasi-real-time multi-dimensional analysis requirements in large-scale business scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the schemes in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0027] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied;

[0028] Figure 2 A flowchart of one embodiment of the data aggregation processing method according to this application is shown;

[0029] Figure 3 It shows Figure 2 A flowchart of one embodiment of step S202;

[0030] Figure 4 It shows Figure 3 A flowchart of one embodiment of step S301;

[0031] Figure 5 A schematic diagram of one embodiment of the data aggregation processing apparatus according to this application is shown;

[0032] Figure 6 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0034] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0036] like Figure 1As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0037] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0038] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0039] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0040] It should be noted that the data aggregation processing method provided in the embodiments of this application is generally executed by a server / terminal device, and correspondingly, the data aggregation processing apparatus is generally located in the server / terminal device.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; the system can have any number of terminal devices, networks, and servers depending on implementation needs.

[0042] Continue to refer to Figure 2 A flowchart illustrating an embodiment of a data aggregation processing method according to this application is shown. The data aggregation processing method includes the following steps:

[0043] S201, extract dimension fields from the source data table, perform preliminary classification of the dimension fields using preset mapping rules, and obtain an initial dimension set;

[0044] Specifically, in the process of extracting dimension fields from the source data table, it is necessary to first identify the core dimensions required for business analysis, such as time, region, product category, and customer type. After extraction, a preset mapping rule is applied to each dimension field to initially classify the original discrete or continuous values. For example, the time dimension can be divided by hour, morning, afternoon, evening, and early morning; the geographic dimension can be grouped by province, city, or region; and the product category can be mapped by category, subcategory, or brand. The preset mapping rules can be provided by business experts or obtained from historical data statistics, and user-defined grouping is also supported. After the initial classification, an initial dimension set is generated, with each dimension field containing its available grouping labels. Technically, the initial dimension set needs to ensure complete coverage of all dimension values ​​in the source data table while supporting efficient indexing and fast access during dynamic aggregation calculations. In a big data environment, the construction of the initial dimension set usually adopts columnar scanning or batch data extraction. If necessary, Bloom filters or dictionary encoding can be used to compress dimension values, thereby reducing memory consumption and I / O pressure.

[0045] It's important to note that when constructing the initial dimension set, the relationships between dimension fields must also be considered. For example, time and product category often require joint analysis, i.e., analyzing the sales performance of various products within different time periods. Therefore, during initial classification, a correlation mapping can be designed to treat time and product category as composite dimensions. Furthermore, for dimension fields with hierarchical relationships, such as provinces, cities, and regions within a geographic dimension, hierarchical indexes can be constructed to flexibly select different granularities of data aggregation during aggregation calculations. Constructing such hierarchical indexes helps to quickly locate specific data ranges, improving data processing efficiency.

[0046] S202, Based on the field attributes in the initial dimension set, apply the hierarchical clustering algorithm to generate a dynamic grouping structure. If the dynamic grouping structure matches the preset standard structure, perform group mapping on the dynamic grouping structure and obtain the dynamic grouping mapping.

[0047] Specifically, dynamically generating a grouping structure based on the field attributes in the initial dimension set requires combining statistical features, data distribution, and hierarchical relationships. For example, for the time dimension, one can first count the number of visits or sales in each hour, and then use a hierarchical clustering algorithm to merge time periods with similar business characteristics into groups, thus obtaining dynamic groups such as morning, afternoon, evening, and early morning. The application of hierarchical clustering algorithms here can adapt to data distribution and avoid sparsity or skewness problems caused by fixed grouping. The generated dynamic grouping structure also needs to be matched with a preset standard structure, such as predefined regional divisions, risk level divisions, or product category stratifications. Matching and mapping are performed through similarity or mapping rules to ensure that the dynamic grouping structure is consistent with the business logic. After a successful match, a dynamic grouping mapping will be generated, including the dynamic grouping label corresponding to each dimension value and its parent-child hierarchical relationship. Technically, methods such as matrix similarity calculation, tree structure traversal, or hash mapping can be used to complete the grouping mapping while retaining traceability information.

[0048] It's important to note that the impact of data updates and changes on the grouping structure must be considered during the construction of dynamic grouping maps. Due to the constantly changing business environment, dimension values ​​in the source data tables may be added or disappear, requiring the dynamic grouping structure to adapt to these changes. Therefore, an incremental update mechanism can be designed to automatically check and update the grouping map when new data records are scanned and assigned to the grouping structure, ensuring the timeliness and accuracy of the grouping structure. Furthermore, for frequently changing dimension fields, a threshold can be set; when the change exceeds this threshold, a re-clustering process is triggered, generating a new dynamic grouping structure. This mechanism helps improve the flexibility and adaptability of data aggregation processing, enabling it to better serve business analysis needs.

[0049] S203, Perform a one-time scan of the source data table based on the dynamic grouping mapping, and assign each scanned data record to the corresponding grouping structure to obtain data partitions;

[0050] Specifically, the core of using dynamic grouping mapping for a one-time scan of the source data table is to quickly assign each data record to the corresponding grouping structure, forming data partitions. In practice, each record is written to its corresponding partition by reading its dimension field value, finding the corresponding group label according to the dynamic grouping mapping, and then writing it to the corresponding partition. For large data volume scenarios, batch scanning or streaming processing can be used, combined with columnar storage or in-memory tables to improve access efficiency. The partition structure can be a multi-level nested hash table or tree structure to support multi-dimensional combination aggregation and reduce redundant calculations. Technically, it is necessary to ensure that each record is accurately mapped to all its corresponding dimension combinations to avoid omissions or duplications. At the same time, to optimize performance, data pruning (filter pushdown) or incremental aggregation (partial aggregation) can be performed during the scan to reduce the amount of computation. The design of data partitions also needs to consider efficient access when building indexes related to metrics, ensuring that each partition can quickly locate the corresponding aggregation metric, achieving one scan and multiple reuses. The core of this step is to complete the source table scan in one go, while accurately generating data partitions that can be directly used for aggregation calculations.

[0051] S204: Extract the indicator value sequence from the data partition, construct the association index between the indicator and the data partition using the hash mapping algorithm, and determine whether the association index covers all dimension combinations. If it covers all, the indexed data structure is obtained.

[0052] Specifically, after extracting the sequence of metric values ​​from the data partitions, it is necessary to build an index linking the metrics to the data partitions. Each metric (such as order quantity, total sales, or number of unique buyers) corresponds to one or more data partitions with different dimensions. A hash mapping algorithm is used to quickly establish the index relationship. The hash mapping generates a unique key based on the dimension combination, mapping the sequence of metric values ​​to the corresponding partition, thus forming a directly accessible indexed data structure. The indexed structure can be a multi-level hash table, a sparse matrix, or a columnar storage format to support efficient aggregation calculations and multi-dimensional queries. After construction, it is necessary to verify the index coverage, i.e., check whether each dimension combination has a corresponding metric value in the index to ensure no missing or omitted values. If missing values ​​exist, coverage can be achieved through default value filling, backtracking data mapping, or supplementary calculations.

[0053] For example, suppose an e-commerce platform needs to aggregate and analyze order data, with dimensions of "user region + purchase time (24 hours) + product category" and metrics of "order quantity, total sales, and number of unique buyers". After data partitioning, the system extracts the sequence of metric values ​​from each partition and builds an index using hash mapping. For instance, if a record's dimension combination is "South China + 8 AM + Electronics", the system will generate a unique hash key (e.g., South China #08# Electronics) and write the three metrics under this combination—order quantity, sales, and number of buyers—into the index structure. This way, queries don't need to backtrack to the original data table; instead, the corresponding metric value is directly located using the hash key, achieving constant time complexity access. If you need to query "order quantity of electronics in South China in the morning", the system will first find the partition corresponding to morning and then quickly retrieve the value using the hash key. Simultaneously, to avoid omissions, the system will iterate through all dimension combinations to ensure that each combination has a metric value; if a value is missing, it will be filled with zero or padded.

[0054] S205, based on the indexed data structure, execute the aggregation calculation function to calculate the sum or average of each dimension combination to obtain the aggregation result set.

[0055] Specifically, when executing aggregation calculation functions based on indexed data structures, the metric values ​​of each dimension combination can be quickly accessed via indexes for summation, averaging, or other aggregation operations. Technically, vectorized computation, block computation, or streaming aggregation methods can be used to improve processing efficiency. For example, for summing order quantities, the metric values ​​within each partition can be accumulated; for calculating total sales, summation can be performed in parallel in memory; and for counting unique purchasing users, a Bloom filter or hash set can be used to remove duplicates before counting. Aggregation calculations require traversing all dimension combinations, but with the help of indexed data structures, the calculation process no longer needs to repeatedly scan the source table or repeatedly match dimensions. For complex business requirements, metric constraints can be further supported, such as performing secondary aggregation based on custom groupings on the original dimension combinations. In technical implementation, distributed computing frameworks (such as Spark and Flink) combined with in-memory tables and parallel algorithms may be used to improve the timeliness and scalability of large-scale data aggregation. The final result set is organized by dimension combination to form a complete aggregation table, which can be directly used by downstream analysis, reporting, or business systems to achieve efficient, multi-dimensional, and scalable data aggregation processing.

[0056] Further, please refer to Figure 3 Based on the field attributes in the initial dimension set, a hierarchical clustering algorithm is applied to generate a dynamic grouping structure. If the dynamic grouping structure matches a preset standard structure, a grouping mapping is performed on the dynamic grouping structure, and the dynamic grouping mapping is obtained. The specific steps include:

[0057] S301, based on the similarity of the dimension fields in the initial dimension set and the business logic requirements, the dimension fields are merged step by step using a hierarchical clustering algorithm to form dynamic grouping structures at different levels. Each level of grouping structure represents the aggregation requirements for different granularities of data.

[0058] S302, For dynamic grouping structures, check whether the grouping structure at each level matches the preset standard structure;

[0059] S303, If the grouping structure matches the standard structure, then the grouping structure is stored;

[0060] S304, establish the mapping between grouping structure and dimension fields in the storage database, and obtain the dynamic grouping mapping.

[0061] In this embodiment, the similarity of fields in the initial dimension set can be measured through statistical features (such as numerical distribution, value range, and variance) and business logic rules (such as geographical region affiliation, time continuity, and risk level distribution). Then, a hierarchical clustering algorithm is applied to gradually merge similar fields, forming a tree-like multi-level grouping structure. For example, in the time dimension, the bottom layer can be the hour level, which can be merged level by level to obtain morning, afternoon, evening, and early morning, and further merged into weekdays and non-working days, thus forming time aggregation requirements of different granularities. In the geographical dimension, the bottom layer is cities, which are gradually merged into provinces, regions, and even the whole country. The clustering algorithm not only considers the similarity between field values, but also needs to combine business semantics. For example, "East China" must include specific provinces. Such rules will be embedded as constraints in the clustering process. After the dynamic grouping structure is generated, each level will be verified to check whether it conforms to the preset standard structure (such as the company's unified regional division rules or risk control level definition). If it matches, the structure will be fixed and stored. After storage, a mapping relationship between the grouping structure and the dimension fields needs to be established in the database to form a dynamic grouping mapping. For example, the time dimension "08:00–11:00" is marked as "AM" in the mapping, which is consistent with the standard grouping required by the business. This mapping not only provides a direct reference for querying and aggregation, but can also be reused when expanding new business dimensions, avoiding redundant construction and improving system scalability and maintainability.

[0062] By following the steps above, we can ensure that the dynamic grouping results not only conform to the data distribution pattern but also meet the business standard definition, thus realizing a flexible, efficient, and reusable multi-dimensional aggregation and grouping scheme.

[0063] Further, please refer to Figure 4 Based on the similarity of the dimension fields in the initial dimension set and the business logic requirements, a hierarchical clustering algorithm is used to merge the dimension fields level by level to form dynamic grouping structures at different levels. Specifically, this includes:

[0064] S401, obtain the association graph of the dimension fields, and extract the association relationship of each dimension field from the association graph;

[0065] S402 quantifies the relationships between the fields of each dimension to obtain the similarity of the fields of each dimension;

[0066] S403, Set the initial parameters for the hierarchical clustering algorithm, including the number of clusters and the similarity threshold;

[0067] S404: Select the most representative dimension field from the initial dimension set as the cluster center. The most representative dimension field is the field in the initial dimension set that best reflects the data characteristics and business needs and has a significant degree of distinction from other dimension fields.

[0068] S405, based on similarity, sequentially assigns each dimension field to a cluster center to form a dynamic grouping structure.

[0069] In this embodiment, the association graph of dimensional fields can be constructed by defining the co-occurrence frequency, statistical distribution similarity, and logical relationships of historical business data. For example, the time dimension has a strong correlation with sales revenue, while the geographical dimension has a more significant correlation with logistics costs. This association information is stored in the form of graph nodes and edges. Based on the association graph, the association relationships between each dimensional field are quantified using methods such as correlation coefficients, mutual information, or Jaccard similarity to obtain a numerical similarity matrix. Next, the initial parameters of the hierarchical clustering algorithm are set. The number of clusters can be determined according to the aggregation granularity requirements of the business scenario. For example, the number of clusters for the time dimension can be set to 4 (morning, afternoon, evening, early morning). The similarity threshold is used to control the strictness of merging to avoid information loss due to excessive aggregation. Subsequently, the most representative dimensional field is selected from the initial dimensional set as the cluster center. This selection process can be based on the field's entropy value, information gain, or business definition weight. For example, the "hour field" can most meticulously reflect business fluctuations in the time dimension, and is therefore often used as a cluster center. Finally, based on the similarity matrix, other dimension fields are sequentially assigned to appropriate cluster centers, generating a dynamic grouping structure that merges at each level. This grouping structure not only follows the statistical patterns of the data but also accommodates the hierarchical aggregation granularity required for business analysis.

[0070] For example, in e-commerce order analysis, the system first extracts dimensional fields from historical order data, such as "purchase time (hours)," "user region (city, province)," and "product category," and constructs a correlation graph. For instance, statistics show a high co-occurrence frequency between "purchase time (8 PM – 11 PM)" and "electronic products," with a significant increase in the corresponding "sales volume," indicating a strong correlation between the two. Simultaneously, "user region (remote provinces)" shows a strong correlation with "logistics costs," demonstrating a close relationship between geographical location and logistics costs. In the correlation graph, "purchase time" and "electronic products" are connected as two nodes by a high-weight edge, while "region" and "logistics costs" are also connected by an edge. Next, the system quantifies the weights of these edges; for example, using mutual information or correlation coefficients, the similarity between "purchase time – electronic products" is calculated to be 0.85, and the similarity between "region – logistics costs" is 0.92. Based on these similarities, and combined with the set clustering threshold, the hierarchical clustering algorithm will gradually merge "purchase time from 8 pm to 11 pm" and "electronic products" into the same group, while merging "remote provinces" and "high logistics costs" into another group, thus forming a dynamic grouping structure that conforms to business logic.

[0071] By following the steps above, hierarchical dynamic grouping of dimension fields can be achieved, which not only conforms to the inherent laws of data but also fits the business logic, ensuring the scientific nature and applicability of the grouping structure.

[0072] Furthermore, the steps of establishing a mapping between grouping structures and dimension fields in the storage database and obtaining dynamic grouping mappings specifically include:

[0073] In the storage database, create a mapping table, which is used to store the correspondence between dynamic grouping structures and dimension fields;

[0074] For each dynamic grouping structure, record the unique identifier of the dynamic grouping structure and the dimensional field information it contains in the mapping table;

[0075] By querying the mapping table, you can quickly locate the dimension field corresponding to any dynamic grouping structure and obtain the dynamic grouping mapping.

[0076] In this embodiment, to achieve an efficient mapping between dynamic grouping structures and dimension fields, the system establishes a dedicated mapping table in the storage database. This mapping table typically includes a unique identifier for the grouping structure (GroupID), grouping hierarchy information (e.g., first-level time grouping, second-level time grouping), a list of corresponding dimension field values ​​(e.g., the hour field's value range: 08–11 AM, mapped to "AM"), and group metadata (e.g., creation time, grouping rule source, version number, etc.). After dynamic grouping is generated, the system writes each grouping structure and its corresponding dimension field into the mapping table one by one. For example, the "AM" grouping structure will be assigned a GroupID = 1001, its dimension field information will be "purchase time field, 08–11 AM", and stored in the mapping table. In this way, all dynamic grouping structures can be permanently stored in the form of unique identifiers and support versioned management, avoiding confusion caused by changes in business rules. When it is necessary to query the specific definition of a grouping structure or quickly locate its corresponding dimension field, simply retrieve the complete mapping relationship by searching the mapping table using the GroupID. This method not only supports single-dimensional mapping but can also be extended to multi-dimensional combined mapping, such as "East China + Morning + Electronic Products". Its mapping table records the value ranges corresponding to multiple dimension fields and corresponds to the GroupID bound to the combination. In this way, dynamic grouping mapping can serve as a unified query entry point.

[0077] By following the steps above, the correspondence between the dynamic grouping structure and the dimension fields can be clearly identified, supporting fast retrieval and reuse, thereby improving the flexibility and reliability of multidimensional aggregation calculations.

[0078] Furthermore, the step of performing a one-time scan of the source data table based on dynamic grouping mapping, assigning each scanned data record to the corresponding grouping structure, and obtaining data partitioning specifically includes:

[0079] Based on the dynamic grouping mapping, data records are sequentially scanned starting from the beginning of the source data table;

[0080] For each scanned data record, the corresponding grouping structure is found in the dynamic grouping mapping based on the dimension field value in the data record;

[0081] Data records are assigned to a matching grouping structure, and a data list is maintained within the matching grouping structure, where the data list is used to store all data records assigned to the matching grouping structure;

[0082] After all data records in the source data table have been scanned and assigned to the corresponding matching grouping structure, several data partitions are obtained, with each data partition corresponding to a grouping structure.

[0083] In this embodiment, the system performs a one-time sequential scan of the source data table based on the dynamic grouping mapping, avoiding the performance overhead of repeated reads. During the scan, each data record has its key dimension field values ​​extracted, such as user region, purchase time, and product category. The system then quickly locates the grouping structure to which the record belongs by looking up the dynamic grouping mapping table. For example, if a data record has the dimension values ​​"East China, 09:00, Clothing", the system will find the corresponding grouping label "East China + Morning + Clothing" in the mapping table and write the record into the data list maintained by that grouping structure. To improve efficiency, the mapping lookup generally uses hash indexes or dictionary encoding, making the matching process close to constant time complexity. As the scan progresses, all records are gradually assigned to the corresponding grouping structures, eventually forming several data partitions, each maintaining a complete set of data records. These partitions not only directly support aggregation calculations but also reduce redundant processing of cross-partition data, ensuring the efficiency and accuracy of the aggregation calculation logic.

[0084] By following the steps above, a one-time scan of the source data table and the automatic allocation of data partitions can be achieved, significantly reducing the overhead of repeated table reads and data allocation, and improving the efficiency and scalability of aggregation processing.

[0085] Furthermore, the steps of extracting indicator value sequences from data partitions, constructing an association index between indicators and data partitions using a hash mapping algorithm, and determining whether the association index covers all dimension combinations, and obtaining the indexed data structure if all are covered, specifically include:

[0086] The indicator value sequence is extracted sequentially from each data partition. The indicator value sequence reflects the measurement information of the data record under different dimensions.

[0087] For the sequence of indicator values, a preset hash mapping algorithm is applied to map each indicator value to a unique hash value in order to build an association index between the indicator and the data partition. The hash value serves as the key of the association index, and the data partition serves as the value of the association index.

[0088] Traverse the associated indexes and check for any uncovered dimension combinations;

[0089] If the associated index covers all dimension combinations, then the indexed data structure of the data partition is obtained.

[0090] In this embodiment, the system first extracts a sequence of business analysis-related metrics from each data partition, such as order quantity, sales revenue, number of unique users, and logistics costs. These metrics reflect the measurement information of data records under different dimensional combinations. Then, for each metric sequence, the system applies a preset hash mapping algorithm to encode the dimensional combination (e.g., "East China + Morning + Electronic Products") into a unique hash key. This key serves as the entry point for the index, and the corresponding data partition is used as the storage object for the hash value. In this way, a one-to-one index structure is formed between metrics and data partitions, allowing direct location of the target partition through the hash key during queries or aggregation calculations, eliminating the need for dimensional matching or repeated scanning. After mapping, the system traverses the entire associated index, checking one by one whether any dimensional combinations have not yet been indexed. For example, if there is no data for a specific region and time period combination, default values ​​need to be added to the index structure to maintain completeness. If all dimensional combinations are confirmed to be covered, the system obtains a complete indexed data structure that supports fast retrieval and parallel aggregation operations.

[0091] By following the steps above, a complete index relationship can be established between metrics and data partitions, achieving full coverage of dimension combinations, thereby supporting rapid location and efficient aggregation calculations.

[0092] Furthermore, based on the indexed data structure, the steps of executing aggregation calculation functions to sum or average the combinations of each dimension to obtain the aggregation result set specifically include:

[0093] Based on the indexed data structure, for each dimension combination, the matching preset aggregation calculation function is called;

[0094] In the aggregation calculation function, the appropriate aggregation operation is selected according to the type of dimension combination. The aggregation operation includes summation or average calculation.

[0095] Traverse the indexed data structure, extract the corresponding data records for each dimension combination, and perform the corresponding aggregation operation to obtain the aggregation operation result;

[0096] The results of the aggregation operation are stored in the aggregation result set, which records the aggregated value of each dimension combination.

[0097] In this embodiment, after obtaining the indexed data structure, the system calls the preset aggregation calculation function for each dimension combination to achieve statistical summarization of different indicators. For example, for indicators such as order quantity, the system automatically selects the summation function; while for indicators such as user dwell time or average spending amount, it selects the average calculation function. In the specific implementation process, the system first traverses the indexed data structure, extracts the corresponding data record set according to the dimension combination (such as "East China + Morning + Electronic Products"), and then passes it to the aggregation calculation function. The aggregation function performs vectorized calculation on the data sequence to avoid the performance overhead of processing each record individually; in a distributed environment, it can also accelerate the processing of large-scale data through partitioned parallel computing and local aggregation followed by merging. The aggregation results of each dimension combination are stored in the result set and organized according to the dimension hierarchy. For example, the aggregation results of each hour under the time dimension can be summarized to higher levels such as morning and afternoon. Finally, the aggregation result set contains the aggregation values ​​of all dimension combinations, which can be directly used as report output or called by upper-level business systems.

[0098] For example, in e-commerce sales data analysis, a certain dimension combination might be "East China + Morning + Electronics," with its corresponding data partition containing several order records. When traversing the indexed data structure, the system extracts the order quantity field for this combination and uses a summation function to accumulate them. For instance, 5 orders placed at 10 AM and 3 orders placed at 11 AM result in a total of 8 orders. Simultaneously, for metrics like spending amount, the system uses an average calculation function to average the amounts of all orders within the same combination (e.g., 100 yuan, 150 yuan, 250 yuan), yielding an average order value of 166.7 yuan for that period. Finally, the system writes these two aggregation results into the aggregation result set, ensuring that the "East China + Morning + Electronics" dimension combination contains both the total number of orders and the average spending amount.

[0099] By following the steps above, computational efficiency can be significantly improved while ensuring the integrity of the aggregation results.

[0100] In the above embodiments, this application discloses a data aggregation processing method, belonging to the field of big data technology, and applied to data aggregation in the financial field. First, by using mapping rules and hierarchical clustering algorithms to group dimensional fields, it can flexibly adapt to different granularity analysis needs while ensuring business logic consistency, avoiding data sparsity or bias caused by unreasonable dimension splitting. Second, by establishing a dynamic grouping mapping and performing a one-time scan on the source data table, each data record is directly assigned to the corresponding grouping structure, reducing the overhead of multiple accesses to the source table and significantly improving data processing efficiency. Simultaneously, by constructing an association index between indicators and data partitions based on a hash mapping algorithm, it can not only quickly locate indicator values ​​under any combination of dimensions but also ensure full coverage of the dimension combination by the index, thereby improving the completeness and accuracy of the results. With the support of the indexed data structure, aggregation calculation functions can efficiently perform operations such as summation and averaging, achieving rapid aggregation of multidimensional data. Overall, this application significantly reduces the computational and storage pressure of the system while ensuring the correctness of the aggregation results, improving the timeliness and scalability of data analysis, and is particularly suitable for real-time or near-real-time multidimensional analysis needs in large-scale business scenarios.

[0101] In this embodiment, the data aggregation processing method runs on an electronic device (e.g., Figure 1 The server shown can receive instructions or acquire data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, Wi-Fi connections, Bluetooth connections, Wi-Fi connections, Zigbee connections, UWB (ultra-Width band) connections, and other currently known or future wireless connection methods.

[0102] It should be emphasized that, to further ensure the privacy and security of the aforementioned source data table information, the source data table information can also be stored in a node of a blockchain.

[0103] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0104] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0105] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0107] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0108] Further reference Figure 5 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data aggregation processing apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0109] like Figure 5 As shown, the data aggregation processing apparatus 500 described in this embodiment includes:

[0110] The dimension classification module 501 is used to extract dimension fields from the source data table, perform preliminary classification of the dimension fields using preset mapping rules, and obtain an initial dimension set.

[0111] The grouping mapping module 502 is used to generate a dynamic grouping structure by applying a hierarchical clustering algorithm based on the field attributes in the initial dimension set. If the dynamic grouping structure matches the preset standard structure, the dynamic grouping structure is grouped and mapped, and the dynamic grouping mapping is obtained.

[0112] The data table scanning module 503 is used to perform a one-time scan of the source data table according to the dynamic grouping mapping, and to assign each scanned data record to the corresponding grouping structure to obtain data partitions;

[0113] The associated index module 504 is used to extract indicator value sequences from data partitions, construct an associated index between indicators and data partitions using a hash mapping algorithm, and determine whether the associated index covers all dimension combinations. If it covers all, an indexed data structure is obtained.

[0114] The aggregation operation module 505 is used to execute aggregation calculation functions based on the indexed data structure to calculate the sum or average of each dimension combination and obtain the aggregation result set.

[0115] Furthermore, the group mapping module 502 specifically includes:

[0116] The hierarchical merging unit is used to merge the dimension fields in the initial dimension set according to the similarity of the dimension fields and the business logic requirements, using a hierarchical clustering algorithm to form dynamic grouping structures at different levels. Each level of grouping structure represents the aggregation requirements for different granularities of data.

[0117] The structure matching unit is used to check whether the grouping structure at each level matches the preset standard structure for dynamic grouping structures.

[0118] A structure storage unit is used to store the group structure if the group structure matches the standard structure.

[0119] The grouping mapping unit is used to establish the mapping between grouping structure and dimension fields in the storage database and to obtain dynamic grouping mappings.

[0120] Furthermore, the step-by-step merging units specifically include:

[0121] The field association sub-unit is used to obtain the association graph of the dimension fields and extract the association relationship of each dimension field from the association graph;

[0122] The correlation quantification subunit is used to quantify the correlation between fields of each dimension and obtain the similarity of each field.

[0123] The parameter setting subunit is used to set the initial parameters of the hierarchical clustering algorithm, including the number of clusters and the similarity threshold.

[0124] The cluster center selection sub-unit is used to select the most representative dimension field from the initial dimension set as the cluster center. The most representative dimension field is the field in the initial dimension set that best reflects the data characteristics and business needs and has a significant degree of distinction from other dimension fields.

[0125] The dimensional classification subunit is used to sequentially assign each dimension field to a cluster center based on similarity, forming a dynamic grouping structure.

[0126] Furthermore, the group mapping unit specifically includes:

[0127] The mapping table creation sub-unit is used to create a mapping table in the storage database, where the mapping table is used to store the correspondence between dynamic grouping structures and dimension fields;

[0128] The information recording sub-unit is used to record the unique identifier of each dynamic grouping structure and the included dimension field information into the mapping table for each dynamic grouping structure.

[0129] The dynamic grouping mapping subunit is used to quickly locate the dimension field corresponding to any dynamic grouping structure by querying the mapping table and obtain the dynamic grouping mapping.

[0130] Furthermore, the data table scanning module 503 specifically includes:

[0131] The sequential scan unit is used to sequentially scan data records starting from the beginning of the source data table according to the dynamic grouping mapping.

[0132] The structure matching unit is used to find the corresponding grouping structure in the dynamic grouping mapping for each scanned data record based on the dimension field values ​​in the data record;

[0133] The data allocation unit is used to allocate data records to the matching group structure and maintain a data list inside the matching group structure, wherein the data list is used to store all data records allocated to the matching group structure;

[0134] A data partitioning unit is used to obtain several data partitions after all data records in the source data table have been scanned and assigned to the corresponding matching grouping structure. Each data partition corresponds to a grouping structure.

[0135] Furthermore, the associated index module 504 specifically includes:

[0136] The indicator acquisition unit is used to extract the indicator value sequence from each data partition in sequence. The indicator value sequence reflects the measurement information of the data record under different dimensions.

[0137] The associated index unit is used to apply a preset hash mapping algorithm to the sequence of indicator values, mapping each indicator value to a unique hash value, in order to build an associated index between the indicator and the data partition. The hash value serves as the key of the associated index, and the data partition serves as the value of the associated index.

[0138] The coverage check unit is used to traverse the associated indexes and check for any uncovered dimension combinations.

[0139] Indexed data structure unit, used to obtain the indexed data structure of data partitions if the associated index covers all dimension combinations.

[0140] Furthermore, the aggregation operation module 505 specifically includes:

[0141] The function call unit is used to call the matching preset aggregation calculation function for each dimension combination based on the indexed data structure;

[0142] The aggregation operation selection unit is used to select the appropriate aggregation operation in the aggregation calculation function according to the type of dimension combination, wherein the aggregation operation includes summation or average calculation;

[0143] The aggregation operation unit is used to traverse the indexed data structure, extract the corresponding data records for each dimension combination, and perform the corresponding aggregation operation to obtain the aggregation operation result.

[0144] The result storage unit is used to store the results of aggregation operations into an aggregation result set, where the aggregation result set records the aggregated value of each dimension combination.

[0145] In the above embodiments, this application discloses a data aggregation processing apparatus, belonging to the field of big data technology, and applied to data aggregation in the financial field. First, by using mapping rules and hierarchical clustering algorithms to group dimensional fields, it can flexibly adapt to different granularity analysis needs while ensuring business logic consistency, avoiding data sparsity or bias caused by unreasonable dimension splitting. Second, by establishing a dynamic grouping mapping and performing a one-time scan on the source data table, each data record is directly assigned to the corresponding grouping structure, reducing the overhead of multiple accesses to the source table and significantly improving data processing efficiency. Simultaneously, by constructing an association index between indicators and data partitions based on a hash mapping algorithm, it can not only quickly locate indicator values ​​under any combination of dimensions but also ensure full coverage of the dimension combination by the index, thereby improving the completeness and accuracy of the results. With the support of the indexed data structure, aggregation calculation functions can efficiently perform operations such as summation and averaging, achieving rapid aggregation of multidimensional data. Overall, this application significantly reduces the computational and storage pressure of the system while ensuring the correctness of the aggregation results, improving the timeliness and scalability of data analysis, and is particularly suitable for real-time or near-real-time multidimensional analysis needs in large-scale business scenarios.

[0146] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0148] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0149] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for data aggregation processing methods. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0150] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or to process data, such as computer-readable instructions for executing the data aggregation processing method.

[0151] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0152] This application also provides an embodiment, namely, a computer device including a memory and a processor. The memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of the data aggregation processing method described above, that is, implements:

[0153] A method for data aggregation processing includes:

[0154] Dimension fields are extracted from the source data table, and the dimension fields are initially classified using preset mapping rules to obtain an initial set of dimensions.

[0155] Based on the field attributes in the initial dimension set, a hierarchical clustering algorithm is applied to generate a dynamic grouping structure. If the dynamic grouping structure matches the preset standard structure, the dynamic grouping structure is grouped and mapped, and the dynamic grouping mapping is obtained.

[0156] Based on the dynamic grouping mapping, a one-time scan of the source data table is performed, and each scanned data record is assigned to the corresponding grouping structure to obtain data partitions;

[0157] Extract the sequence of indicator values ​​from the data partitions, construct the association index between the indicators and the data partitions using a hash mapping algorithm, and determine whether the association index covers all dimension combinations. If it covers all, then the indexed data structure is obtained.

[0158] Based on the indexed data structure, the aggregation calculation function is executed to calculate the sum or average of each dimension combination, resulting in an aggregated result set.

[0159] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the data aggregation processing method described above, i.e., to implement:

[0160] A method for data aggregation processing includes:

[0161] Dimension fields are extracted from the source data table, and the dimension fields are initially classified using preset mapping rules to obtain an initial set of dimensions.

[0162] Based on the field attributes in the initial dimension set, a hierarchical clustering algorithm is applied to generate a dynamic grouping structure. If the dynamic grouping structure matches the preset standard structure, the dynamic grouping structure is grouped and mapped, and the dynamic grouping mapping is obtained.

[0163] Based on the dynamic grouping mapping, a one-time scan of the source data table is performed, and each scanned data record is assigned to the corresponding grouping structure to obtain data partitions;

[0164] Extract the sequence of indicator values ​​from the data partitions, construct the association index between the indicators and the data partitions using a hash mapping algorithm, and determine whether the association index covers all dimension combinations. If it covers all, then the indexed data structure is obtained.

[0165] Based on the indexed data structure, the aggregation calculation function is executed to calculate the sum or average of each dimension combination, resulting in an aggregated result set.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0167] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0168] It should be noted that the software tools or components not belonging to this company that appear in the various embodiments of this application are merely illustrative examples and do not represent actual use.

[0169] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method of data aggregation processing, characterized by, The method comprises the following steps: Extracting dimension fields from a source data table, and preliminarily classifying the dimension fields according to preset mapping rules to obtain an initial dimension set; According to the field attributes in the initial dimension set, applying a hierarchical clustering algorithm to generate a dynamic grouping structure, and if the dynamic grouping structure matches a preset standard structure, performing grouping mapping on the dynamic grouping structure and obtaining a dynamic grouping mapping; According to the dynamic grouping mapping, performing one-time scanning on the source data table, distributing each scanned data record to the corresponding grouping structure, and obtaining data partitions; Extracting index value sequences from the data partitions, constructing an association index of the index and the data partitions by using a hash mapping algorithm, and judging whether the association index covers all dimension combinations, and if all dimension combinations are covered, obtaining an indexed data structure; According to the indexed data structure, performing an aggregation calculation function to calculate the sum or average value of each dimension combination to obtain an aggregation result set.

2. The method of data aggregation processing of claim 1, wherein, The step of generating a dynamic grouping structure according to the field attributes in the initial dimension set and applying a hierarchical clustering algorithm, and if the dynamic grouping structure matches a preset standard structure, performing grouping mapping on the dynamic grouping structure and obtaining a dynamic grouping mapping, specifically comprises the following steps: According to the similarity of each dimension field in the initial dimension set and the business logic requirement, the hierarchical clustering algorithm is used to merge the dimension fields step by step to form the dynamic grouping structure at different levels, wherein each level of the grouping structure represents different granularity of data aggregation requirement; For the dynamic grouping structure, it is checked whether each level of the grouping structure matches the preset standard structure; If the grouping structure matches the standard structure, the grouping structure is stored; In the storage database, the mapping between the grouping structure and the dimension field is established, and the dynamic grouping mapping is obtained.

3. The method of data aggregation processing of claim 2, wherein, The step of using a hierarchical clustering algorithm to merge the dimension fields step by step to form the dynamic grouping structure at different levels according to the similarity of each dimension field in the initial dimension set and the business logic requirement, specifically comprises the following steps: An association graph of the dimension fields is obtained, and the association relationship of each dimension field is extracted from the association graph; The association relationship of each dimension field is quantified to obtain the similarity of each dimension field; The initial parameters of the hierarchical clustering algorithm are set, wherein the initial parameters include the number of clusters and the similarity threshold; The most representative dimension field is selected from the initial dimension set as the cluster center, wherein the most representative dimension field refers to the field that can best reflect the data characteristics and business requirements in the initial dimension set, and has a significant degree of differentiation from other dimension fields; Based on the similarity, each dimension field is gradually assigned to the cluster center in turn to form the dynamic grouping structure.

4. The method of data aggregation processing of claim 2, wherein, The step of establishing the mapping between the grouping structure and the dimension field in the storage database and obtaining the dynamic grouping mapping, specifically comprises the following steps: In the storage database, a mapping table is created, wherein the mapping table is used to store the correspondence between the dynamic grouping structure and the dimension field; For each dynamic grouping structure, the unique identifier of the dynamic grouping structure and the contained dimension field information are recorded in the mapping table; By querying the mapping table, the corresponding dimension field of any dynamic grouping structure is quickly located, and the dynamic grouping mapping is obtained.

5. The method of data aggregation processing of claim 1, wherein, The step of performing a one-time scan on the source data table according to the dynamic grouping mapping, and distributing each scanned data record to the corresponding grouping structure to obtain data partitions, specifically includes: According to the dynamic grouping mapping, sequentially scan data records from the starting position of the source data table; For each scanned data record, according to the dimension field value in the data record, find the corresponding grouping structure in the dynamic grouping mapping; Distribute the data record to the matching grouping structure, and maintain a data list inside the matching grouping structure, wherein the data list is used to store all data records distributed to the matching grouping structure; After all data records in the source data table are scanned and distributed to the corresponding matching grouping structure, a plurality of data partitions are obtained, wherein each data partition corresponds to a grouping structure.

6. The method of data aggregation processing of claim 1, wherein, Extract the index value sequence from the data partition, use the hash mapping algorithm to construct the association index of the index and the data partition, and judge whether the association index covers all dimension combinations. If all are covered, the step of obtaining the indexed data structure includes: From each of the data partitions, the index value sequence is extracted in turn, wherein the index value sequence reflects the measurement information of the data record under different dimensions; For the index value sequence, a preset hash mapping algorithm is applied to map each index value to a unique hash value to construct the association index of the index and the data partition, wherein the hash value is used as the key of the association index, and the data partition is used as the value of the association index; Traverse the association index to check whether there is an uncovered dimension combination; If the association index covers all dimension combinations, the indexed data structure of the data partition is obtained.

7. The method of data aggregation processing of claim 1, wherein, The step of performing an aggregation calculation function on each dimension combination to obtain a set of aggregation results according to the indexed data structure includes: According to the indexed data structure, for each dimension combination, a matching preset aggregation calculation function is called; In the aggregation calculation function, according to the type of dimension combination, a corresponding aggregation operation is selected, wherein the aggregation operation includes summation or average value calculation; Traverse the indexed data structure, for each dimension combination, extract the corresponding data record, and perform the corresponding aggregation operation to obtain the aggregation operation result; The aggregation operation result is stored in the aggregation result set, wherein the aggregation result set records the aggregation value of each dimension combination.

8. An apparatus for data aggregation processing, the apparatus comprising: It includes: The dimension classification module is configured to extract dimension fields from the source data table, perform preliminary classification on the dimension fields by using a preset mapping rule, and obtain an initial dimension set; The grouping mapping module is configured to generate a dynamic grouping structure by using a hierarchical clustering algorithm according to field attributes in the initial dimension set, perform grouping mapping on the dynamic grouping structure if the dynamic grouping structure matches a preset standard structure, and obtain a dynamic grouping mapping; The data table scanning module is configured to perform one-time scanning on the source data table according to the dynamic grouping mapping, distribute each data record obtained by scanning into a corresponding grouping structure, and obtain a data partition; The association index module is configured to extract a sequence of index values from the data partition, construct an association index between the index and the data partition by using a hash mapping algorithm, and determine whether the association index covers all dimension combinations, and obtain an indexed data structure if all dimension combinations are covered. The aggregation operation module is configured to execute an aggregation calculation function to calculate a sum or an average value for each dimension combination according to the indexed data structure, and obtain an aggregation result set.

9. A computer device, comprising: The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the data aggregation processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the data aggregation processing method according to any one of claims 1 to 7.

Images