Multi-dimensional multi-index data processing method and device, storage medium and electronic equipment

Abstract

The invention provides a multi-dimensional multi-index data processing method and device, a storage medium and electronic equipment, and is applied to the technical field of big data processing. When a program is started, multiple groups of homologous calculation requests are analyzed into an index calculation logic set, a dimension aggregation combination and a filter combination, packaged into calculation combination objects and numbered. And processing data by using the objects, and outputting composite key value pair structure data containing a number ID and a data dimension. And performing aggregation according to the data dimensions of the composite key value pairs, and outputting an aggregation result with a serial number ID by using logic processing data of the calculation combination object. According to the method, the ID for calculating the dimension combination is introduced, and innovative design is carried out in multiple data processing stages, so that the calculation efficiency can be remarkably improved, the calculation process is simplified, and the flexibility of processing complex multi-dimension combination and multi-index calculation is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a multi-dimensional and multi-index data processing method, device, storage medium and electronic device. Background Art

[0002] In the computing scenario of processing massive data, the current mainstream computing engines on the market, such as MySQL, Hadoop and Spark, have significant advantages in data processing and computing capabilities, but still have significant limitations when processing complex multi-dimensional combinations and multi-index calculations. Specifically, the syntax of these computing engines such as GROUP BY ROLLUP / CUBE / SETS does not support batch execution of multi-dimensional combinations and multi-index calculations, making it difficult to perform complex aggregation and analysis on large-scale data.

[0003] Based on this, how to achieve multi-dimensional and multi-indicator data processing to improve computing efficiency and flexibility has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of the above problems, the present invention provides a multi-dimensional multi-index data processing method, device, storage medium and electronic device that overcome the above problems or at least partially solve the above problems. The technical solution is as follows:

[0005] A multi-dimensional and multi-indicator data processing method, comprising:

[0006] In the program startup phase, multiple groups of computing requests from the same source are abstractly parsed into indicator computing logic sets, dimension aggregation combinations, and filter combinations, and the indicator computing logic sets, dimension aggregation combinations, and filter combinations corresponding to each group of computing requests are encapsulated into a computing combination object; each computing combination object is numbered to generate mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the computing combination object;

[0007] Traversing each of the mapping structure data, using the calculation rule description of the calculation combination object in the mapping structure data to process the data, and outputting composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result;

[0008] Each distributed working node performs data aggregation according to the data dimensions in the composite key-value pair structure data, and processes the data using the computational logic of the computational combination object that has a mapping relationship with the number ID in the composite key-value pair structure data, and outputs a data aggregation result, wherein the data aggregation result includes the number ID.

[0009] Optionally, before each distributed working node aggregates data according to the data dimension in the composite key-value pair structure data, and processes data using the computing logic of the computing combination object that has a mapping relationship with the serial number ID in the composite key-value pair structure data, and outputs the data aggregation result, the method further includes:

[0010] The local working node performs data pre-aggregation according to the data dimensions in the composite key-value pair structure data, and processes local data using the computational logic of the computational combination object that has a mapping relationship with the serial number ID in the composite key-value pair structure data, and outputs a data pre-aggregation result, wherein the data pre-aggregation result includes the serial number ID.

[0011] Optionally, the calculation logic includes a general expression calculation library and a custom calculation function, wherein the custom calculation function completes object instantiation and high-performance calling through reflection.

[0012] Optionally, the method further includes:

[0013] The Roaring BitMap algorithm is introduced to integrate deduplication statistics into the bitmap data results, and data pre-aggregation is completed by performing sum and difference calculations on the bitmap.

[0014] Optionally, the method further includes:

[0015] During the program startup phase, the mapping structure data is stored in global variables and distributed computing contexts, so that the driver program and each working node can obtain the mapping structure data.

[0016] A multi-dimensional and multi-index data processing device comprises: an abstract parsing unit, a number mapping unit, a traversal data processing unit and a data aggregation unit.

[0017] The abstract parsing unit is used to abstractly parse multiple groups of computing requests of the same source into an indicator computing logic set, a dimension aggregation combination and a filter combination respectively during the program startup phase, and encapsulate the indicator computing logic set, the dimension aggregation combination and the filter combination corresponding to each group of computing requests into a computing combination object respectively;

[0018] The number mapping unit is used to number each calculation combination object and generate mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the calculation combination object;

[0019] The traversal data processing unit is used to traverse each of the mapping structure data, use the calculation rule description of the calculation combination object in the mapping structure data to process the data, and output composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result;

[0020] The data aggregation unit is used to perform data aggregation according to the data dimensions in the composite key-value pair structure data at each distributed working node, and use the computing logic of the computing combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process the data, and output a data aggregation result, wherein the data aggregation result includes the number ID.

[0021] Optionally, the device further comprises: a data pre-aggregation unit,

[0022] A data pre-aggregation unit is used to perform data pre-aggregation at a local working node according to the data dimension in the composite key-value pair structure data, and use the calculation logic of the calculation combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process local data, and output a data pre-aggregation result, wherein the data pre-aggregation result includes the number ID.

[0023] Optionally, the device further comprises: a data storage unit,

[0024] The data storage unit is used to store the mapping structure data in global variables and distributed computing contexts during the program startup phase, so that the driver program and each working node can obtain the mapping structure data.

[0025] A computer-readable storage medium stores a program, which, when executed by a processor, implements any of the multi-dimensional and multi-index data processing methods described above.

[0026] An electronic device, comprising at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other via the bus; and the processor is used to call program instructions in the memory to execute any of the multi-dimensional and multi-indicator data processing methods described above.

[0027] By means of the above technical scheme, the multi-dimensional and multi-indicator data processing method, device, storage medium and electronic device provided by the present invention, in the program startup phase, abstractly parse multiple groups of computing requests of the same source into indicator computing logic sets, dimension aggregation combinations and filter combinations respectively, and encapsulate the indicator computing logic sets, dimension aggregation combinations and filter combinations corresponding to each group of computing requests into a computing combination object respectively; number each computing combination object and generate mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the computing combination object; traverse each mapping structure data, use the computing rule description of the computing combination object in the mapping structure data to process the data, and output composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result; each distributed working node aggregates data according to the data dimension in the composite key-value pair structure data, and uses the computing logic of the computing combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process the data, and outputs the data aggregation result, wherein the data aggregation result contains the number ID. The present invention introduces the ID of the calculation dimension combination and performs innovative design in multiple data processing stages, which can significantly improve the calculation efficiency and simplify the calculation process, and ultimately achieve the goal of outputting all calculation results with only one calculation. It not only reduces the consumption of computing resources, but also improves the flexibility of processing complex multi-dimensional combinations and multi-indicator calculations.

[0028] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0030] Figure 1 A flowchart of an implementation method of a multi-dimensional and multi-index data processing method provided by an embodiment of the present invention is shown;

[0031] Figure 2 A flowchart of another implementation of the multi-dimensional and multi-index data processing method provided by an embodiment of the present invention is shown;

[0032] Figure 3 A schematic diagram of the structure of a multi-dimensional and multi-index data processing device provided by an embodiment of the present invention is shown;

[0033] Figure 4 Another structural schematic diagram of a multi-dimensional multi-index data processing device provided by an embodiment of the present invention is shown;

[0034] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0036] In the scenario of massive data computing, the amount of data that needs to be processed can often reach 1 billion or even 10 billion levels. A single job may consume hundreds of computing units (CUs) and need to transmit terabytes of data. It is common for the running time to exceed 5 minutes. With the diversification of statistical requirements and the increase in dimensional combinations, the number of computing jobs has also increased. In this case, the execution of the overall job becomes time-consuming, labor-intensive and inflexible, resulting in extremely high computing costs.

[0037] MapReduce is a programming model specifically designed for parallel computing of large-scale data sets (usually larger than 1TB). Its core concepts include "Map" and "Reduce", which draw on the characteristics of functional programming languages ​​and vector programming languages. The MapReduce model greatly facilitates programmers to run their programs on distributed systems without mastering distributed programming.

[0038] The current common MapReduce logic in the industry mainly includes the following points:

[0039] Map process: The Map function processes a single piece of data and outputs a KV type data combination.

[0040] Combiner stage: When the combining ratio is met, aggregation calculation is performed locally to reduce network transmission volume.

[0041] After Shuffle: Reducer collects data based on keys and performs aggregation calculations.

[0042] However, this calculation mode can only calculate one set of calculation dimensions at a time. In actual operation, each set of calculations needs to be run separately, and each combination is calculated once. Therefore, existing big data departments usually adopt a daily calculation method and run jobs for several hours at night. Once a job fails, it needs to be handled in time, otherwise it may cause a large number of calculation delays the next day. This not only requires a lot of manpower for monitoring and maintenance, but also consumes a lot of computing resources, which is extremely costly.

[0043] Therefore, the mainstream computing engines currently on the market still have significant limitations when processing multi-dimensional combinations and multi-index calculations, especially in the scenario of massive data calculations, where computing jobs are time-consuming, costly, and inflexible. As a basic computing model, MapReduce is widely used in massive computing models, but its existing logic can only calculate one set of computing dimensions at a time, resulting in each set of calculations requiring a separate job to be run. Faced with increasingly complex statistical needs and massive data processing needs, the existing computing model seems to be increasingly unable to cope with the situation, and new solutions are urgently needed to improve computing efficiency and flexibility and reduce costs.

[0044] Based on this, in an embodiment of the present invention, a multi-dimensional and multi-indicator data processing method is provided, which performs feature abstraction by analyzing and disassembling the calculation process, and innovatively merges multiple dimension combinations and multi-indicator calculation jobs into one job calculation to save calculation cost and time. Business analysts configure various calculation reports through the system, and data analysts write various SQLs. No matter how many combinations or forms are configured, the calculation process can be abstracted and parsed into four meta-objects: indicator calculation logic set, dimension aggregation combination, filter combination and data source description. For the calculation of the same data source, based on the MapReduce calculation mode, the ID identification dimension of the calculation dimension combination is introduced to abstract the above elements. In the preprocessing, Mapper, Combiner and Reducer stages, through innovative design, the goal of outputting all calculation results through one calculation is finally achieved.

[0045] like Figure 1 As shown, a flowchart of an implementation of a multi-dimensional and multi-index data processing method provided by an embodiment of the present invention is provided. The method may include:

[0046] S100. In the program startup phase, multiple groups of computing requests from the same source are abstractly parsed into indicator computing logic sets, dimension aggregation combinations, and filter combinations, and the indicator computing logic sets, dimension aggregation combinations, and filter combinations corresponding to each group of computing requests are encapsulated into a computing combination object; each computing combination object is numbered and mapping structure data is generated, wherein the mapping structure data contains a mapping relationship between the number ID and the computing combination object.

[0047] Among them, multiple groups of calculation requests from the same source refer to multiple independent but related calculation tasks performed on the same data source or data set. Different calculation requests may involve different calculation indicators, dimension combinations, and filter conditions, but they are all based on the same basic data. By integrating these calculation requests together, the calculation process can be optimized, repeated processing can be reduced, and resources and time can be saved. For example: for a large sales data set, there may be multiple calculation requests, such as: calculating the total sales of each product, sales data grouped by region, and sales trends grouped by time period. Although the specific calculation logic and goals of these calculation requests are different, they all originate from the same sales data set.

[0048] Among them, a measure is a value used to measure or measure a specific feature. A measure calculation logic set refers to a set of logic or formulas used to calculate these values. For example, a measure can be total sales, average order amount, number of users, time spent, number of errors, etc. These indicators may be calculated by summing, averaging, counting, etc.

[0049] Dimensions are attributes used to classify or group data. Dimension aggregation refers to a set of dimensions used to group or aggregate data. For example, dimensions can be time (year, month, day), geographic location (country, region, city), product category, network type, etc. Through these dimensions, data can be sliced ​​and aggregated, for example, viewing sales by month and number of users by region.

[0050] Among them, filters are used to filter data so that only data that meets specific conditions is processed. Filter combinations refer to a set of conditions or rules used to filter data. For example: date range (such as January 2023 to December 2023), product category (such as electronics, clothing), geographic location (such as North America only), etc. By applying these filters, the scope of the data set can be limited to make the analysis more targeted.

[0051] Specifically, the embodiment of the present invention can abstract and parse multiple groups of computing requests of the same source to form three main parts: indicator computing logic set, dimension aggregation combination and filter combination. These parts are encapsulated into a computing combination object (calObjec), for example: calObject (Measures, Dimensions, Filters).

[0052] For ease of understanding, an example is given here to illustrate: assuming that among multiple groups of computing requests from the same source, there is a computing request SQL "select sum(meaa),sum(meab)from tableAwhere dimd>0group by dime,dimf", then the abstractly parsed Measures are "sum(meaa),sum(meab)", Dimensions are "group bydime,dimf", and Filters are "where dimd>0". The embodiment of the present invention, through abstract parsing of computing requests, helps to more conveniently adjust and combine the indicator computing logic set, dimension aggregation combination, and filter combination corresponding to each group of computing requests into a computing combination object, so that large-scale data can be efficiently processed and analyzed later.

[0053] The embodiment of the present invention assigns a serial number ID to any computing combination object, and constructs a mapping relationship between the serial number ID and the computing combination object, generating a mapping structure data "calObjectMAP (serial number ID, calObject)".

[0054] The embodiment of the present invention abstracts and designs objects for the general computing process, forming the following three key components: a logical set of indicator calculations, a dimension aggregation combination, and a filter combination, and expands and innovates the MapReduce (MR) computing model. On the basis of the traditional K (dimension) and V (indicator), the concept of computing dimension combination ID (number ID) is introduced. This extension enables a single calculation to support not only a single dimension combination, but also a multi-dimensional combination. At the same time, the data structure form of "calObjectMAP (number ID, calObject)" is also conducive to improving the access performance of metadata, thereby meeting the computing needs of multi-dimensional combinations.

[0055] S110, traverse each mapping structure data, use the calculation rule description of the calculation combination object in the mapping structure data to process the data, and output the composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result.

[0056] The calculation rule description refers to combining the indicator calculation logic set, dimension aggregation combination and filter combination in the calculation combination object to define and execute specific data calculation tasks.

[0057] Defining a calculation task includes: specifying the indicators (Measures) that need to be calculated, specifying the dimensions (Dimensions) according to which data is grouped and aggregated, and specifying the conditions for data filtering (Filters).

[0058] The calculation task includes: first, applying filters to filter the data set. Then, grouping the filtered data according to the specified dimensions. Finally, applying aggregation functions to the grouped data to calculate the indicators.

[0059] Specifically, in the Mapper stage, the embodiment of the present invention traverses the calculation combination object calObjectMAP, takes out the ID and the corresponding calObject in each calObjectMAP, and processes each data according to the calculation rule description in the calObject. Assuming that there are 1 billion data sources and 10 groups of calculation rules, the Mapper stage will perform 10 billion calculations.

[0060] After processing the data using the calculation rule description of the calculation combination object in the mapping structure data, the embodiment of the present invention merges the number ID into the key to form and output the composite key-value pair structure data "(number ID, key), value)", wherein this key follows the key definition in the original MapReduce rule, is generated according to the combination dimension (Dimensions) in the calculation rule description, and is used to obtain the value of each piece of data.

[0061] S120. Each distributed working node aggregates data according to the data dimensions in the composite key-value pair structure data, and processes the data using the computational logic of the computational combination object that has a mapping relationship with the serial number ID in the composite key-value pair structure data, and outputs the data aggregation result, wherein the data aggregation result includes the serial number ID.

[0062] Specifically, in the reducer stage of the embodiment of the present invention, each distributed working node (distributed worker) aggregates data according to the key, and the calculation requests corresponding to the same key are aggregated together. The corresponding ID is then retrieved using the key, and the indicator calculation logic set (Measures) in the corresponding calculation combination object is found to process the data. For example, when Measures is sum(meaa), the value of meaa is summed and output. After the aggregation calculation is completed, the data aggregation result containing the ID is output. In this way, one job completes calculations of multiple aggregation types.

[0063] The multi-dimensional and multi-indicator data processing method provided by the present invention, in the program startup phase, abstractly parses multiple groups of computing requests of the same source into indicator computing logic sets, dimension aggregation combinations and filter combinations respectively, and encapsulates the indicator computing logic sets, dimension aggregation combinations and filter combinations corresponding to each group of computing requests into a computing combination object respectively; numbers each computing combination object, generates mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the computing combination object; traverses each mapping structure data, uses the computing rule description of the computing combination object in the mapping structure data to process data, and outputs composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result; each distributed working node aggregates data according to the data dimension in the composite key-value pair structure data, and uses the computing logic of the computing combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process data, and outputs the data aggregation result, wherein the data aggregation result contains the number ID. The present invention introduces the ID of the calculation dimension combination and performs innovative design in multiple data processing stages, which can significantly improve the calculation efficiency and simplify the calculation process, and ultimately achieve the goal of outputting all calculation results with only one calculation. It not only reduces the consumption of computing resources, but also improves the flexibility of processing complex multi-dimensional combinations and multi-indicator calculations.

[0064] Optional, based on Figure 1 The method shown, such as Figure 2 As shown, a flowchart of another implementation of the multi-dimensional multi-index data processing method provided by an embodiment of the present invention is shown. Before step S120, the method may further include:

[0065] S200. The local working node pre-aggregates data according to the data dimensions in the composite key-value pair structure data, and processes the local data using the computational logic of the computational combination object that has a mapping relationship with the serial number ID in the composite key-value pair structure data, and outputs the data pre-aggregation result, wherein the data pre-aggregation result includes the serial number ID.

[0066] Specifically, in the combiner stage of the embodiment of the present invention, each local working node (local worker) can pre-aggregate its own local data according to the data dimensions in the composite key-value pair structure data, and the calculation requests corresponding to the same key will be aggregated together, and then the key will be used to retrieve the corresponding number ID, and the indicator calculation logic set (Measures) in the corresponding calculation combination object will be found to process the local data.

[0067] The embodiment of the present invention performs local pre-aggregation of intermediate data generated in the Mapper stage through local working nodes, which can significantly reduce the amount of data that needs to be transmitted to the Reducer stage, thereby optimizing network bandwidth and computing efficiency.

[0068] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, the calculation logic may include a general expression calculation library and a custom calculation function, wherein the custom calculation function completes object instantiation and high-performance calling through reflection.

[0069] Among them, the general expression calculation library is a tool library used to parse and calculate expressions. The general expression calculation library supports multiple operators and functions and can handle complex mathematical and logical operations.

[0070] Among them, the custom calculation function is the calculation logic defined by the user according to specific needs.

[0071] Specifically, the calObject's indicator calculation logic set (Measures) can be described as a variable calculation expression and calculated using the industry's commonly used expression calculation library. For custom calculation functions, the object is instantiated through the reflection mechanism and efficiently called during execution. This method avoids the need for each custom calculation logic to be hard-coded into the code, thereby providing basic support for the configuration of calculation logic.

[0072] Optional, in the above Figure 2 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, the method may further include:

[0073] The Roaring BitMap algorithm is introduced to integrate deduplication statistics into the bitmap data results, and data pre-aggregation is completed by performing sum and difference calculations on the bitmap.

[0074] Specifically, the embodiment of the present invention can introduce the Roaring BitMap library, write a custom Combiner class in the MapReduce job, and use the Roaring BitMap algorithm to perform deduplication statistics.

[0075] General computing models usually only support operators that follow the associative law, such as sum and product. However, these models are not applicable to deduplication counts (e.g. distinct count). Based on this, the embodiment of the present invention introduces the Roaring BitMap algorithm to integrate deduplication statistics into a bitmap data result. In this way, the bitmap can be calculated for sum, difference, etc., so that the pre-aggregation calculation of all operators can be completed in the combiner stage, thereby effectively reducing the amount of data in the Shuffle stage.

[0076] It should be noted that the Shuffle phase refers to the process of redistributing data from the output of one task to the input of another task in distributed computing. In other words, the Shuffle phase is the process of redistributing and transmitting data between different nodes. The Shuffle phase usually follows the Mapper phase or some intermediate calculations and is performed before the Reduce phase.

[0077] Optional, in the above Figure 1 On the basis of one or more corresponding embodiments, in another optional embodiment provided by the embodiment of the present invention, the method may further include:

[0078] During the program startup phase, the mapping structure data is stored in global variables and distributed computing contexts so that the driver program and each working node can obtain the mapping structure data.

[0079] Among them, global variables refer to variables whose values ​​can be accessed and modified in all functions and modules during the operation of the driver. In a distributed computing environment, global variables are usually used to store data that needs to be shared between different tasks and nodes.

[0080] Among them, the distributed computing context (such as Hadoop's JobContext or Spark's SparkContext) is an environment for managing and coordinating distributed computing tasks. It provides a mechanism to share data and configuration parameters throughout the distributed system. The context not only contains the configuration information of the job, but is also responsible for managing the execution of the job, resource scheduling, and task allocation.

[0081] Specifically, when the program starts, you can initialize the MAP mapping structure data by reading the configuration file or database and store it in a global variable. For example, in Spark, you can use the broadcast mechanism to broadcast data to all nodes. Store the MAP mapping structure data in the distributed computing context so that all tasks can access the data through the context. For example: in Hadoop, you can store data in the Configuration object, which will be passed to each task through JobContext when the task starts. In Spark, data can be broadcast to all executors through the SparkContext.broadcast method, so that it can be shared among various tasks.

[0082] The embodiment of the present invention stores the mapping structure data in the global variables and the distributed computing context during the program startup phase, so that the driver program and each working node can obtain the data for use in subsequent computing.

[0083] Although operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0084] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0085] Corresponding to the above method embodiment, the embodiment of the present invention also provides a multi-dimensional multi-index data processing device, whose structure is as follows: Figure 3 As shown, it may include: an abstract parsing unit 10 , a number mapping unit 20 , a traversal data processing unit 30 and a data aggregation unit 40 .

[0086] The abstract parsing unit 10 is used to abstractly parse multiple groups of computing requests of the same source into indicator computing logic sets, dimension aggregation combinations and filter combinations during the program startup phase, and to encapsulate the indicator computing logic sets, dimension aggregation combinations and filter combinations corresponding to each group of computing requests into a computing combination object.

[0087] The number mapping unit 20 is used to number each calculation combination object and generate mapping structure data, wherein the mapping structure data includes a mapping relationship between the number ID and the calculation combination object.

[0088] The traversal data processing unit 30 is used to traverse each mapping structure data, use the calculation rule description of the calculation combination object in the mapping structure data to process the data, and output composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result.

[0089] The data aggregation unit 40 is used to aggregate data according to the data dimensions in the composite key-value pair structure data at each distributed working node, and use the computing logic of the computing combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process the data, and output the data aggregation result, wherein the data aggregation result includes the number ID.

[0090] Optional, based on Figure 3 The device shown, such as Figure 4 As shown, another structural schematic diagram of the multi-dimensional multi-indicator data processing device provided by an embodiment of the present invention, the multi-dimensional multi-indicator data processing device may further include: a data pre-aggregation unit 50.

[0091] The data pre-aggregation unit 50 is used to perform data pre-aggregation at the local working node according to the data dimensions in the composite key-value pair structure data, and use the calculation logic of the calculation combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process the local data, and output the data pre-aggregation result, wherein the data pre-aggregation result includes the number ID.

[0092] Optionally, the calculation logic includes a general expression calculation library and a custom calculation function, wherein the custom calculation function completes object instantiation and high-performance calling through reflection.

[0093] Optionally, the Roaring BitMap algorithm is introduced to integrate deduplication statistics into the bitmap data results, and data pre-aggregation is completed by performing sum and difference calculations on the bitmap.

[0094] Optionally, the multi-dimensional and multi-indicator data processing device may further include: a data storage unit.

[0095] The data storage unit is used to store the mapping structure data in the global variables and distributed computing context during the program startup phase, so that the driver program and each working node can obtain the mapping structure data.

[0096] The multi-dimensional and multi-indicator data processing device provided by the present invention, in the program startup phase, abstractly parses multiple groups of computing requests of the same source into indicator computing logic sets, dimension aggregation combinations and filter combinations respectively, and encapsulates the indicator computing logic sets, dimension aggregation combinations and filter combinations corresponding to each group of computing requests into a computing combination object respectively; numbers each computing combination object and generates mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the computing combination object; traverses each mapping structure data, uses the computing rule description of the computing combination object in the mapping structure data to process data, and outputs composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result; each distributed working node aggregates data according to the data dimension in the composite key-value pair structure data, and uses the computing logic of the computing combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process data, and outputs the data aggregation result, wherein the data aggregation result contains the number ID. The present invention introduces the ID of the calculation dimension combination and performs innovative design in multiple data processing stages, which can significantly improve the calculation efficiency and simplify the calculation process, and ultimately achieve the goal of outputting all calculation results with only one calculation. It not only reduces the consumption of computing resources, but also improves the flexibility of processing complex multi-dimensional combinations and multi-indicator calculations.

[0097] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0098] The multi-dimensional and multi-indicator data processing device includes a processor and a memory. The above-mentioned abstract parsing unit 10, number mapping unit 20, traversal data processing unit 30 and data aggregation unit 40 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0099] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set. By adjusting the kernel parameters, when the program starts, multiple groups of calculation requests from the same source are parsed into a logical set of indicator calculations, a dimension aggregation combination, and a filter combination, and encapsulated into a calculation combination object and numbered. These objects are used to process data and output composite key-value pair structure data containing a number ID and data dimensions. Aggregation is performed according to the data dimension of the composite key-value pair, and the data is processed using the logic of the calculation combination object, and the aggregation result with a number ID is output.

[0100] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the multi-dimensional and multi-indicator data processing method is implemented.

[0101] An embodiment of the present invention provides a processor, which is used to run a program, wherein the multi-dimensional and multi-index data processing method is executed when the program is running.

[0102] like Figure 5 As shown, an embodiment of the present invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003; the processor 1001 is used to call the program instructions in the memory 1002 to execute the above-mentioned multi-dimensional multi-index data processing method. The electronic device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0103] The present invention also provides a computer program product, which, when executed on an electronic device, is suitable for executing a program that initializes the steps of a multi-dimensional and multi-index data processing method.

[0104] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] In a typical configuration, an electronic device includes one or more processors (CPU), a memory, and a bus. The electronic device may also include an input / output interface, a network interface, and the like.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0107] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0108] In the description of the present invention, it should be understood that the terms "up", "down", "front", "back", "left" and "right" etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the positions or elements referred to must have specific directions, be constructed and operate in specific directions. Therefore, they should not be understood as limitations of the present invention.

[0109] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0110] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0111] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A multi-dimensional and multi-index data processing method, characterized in that: include: In the program startup phase, multiple groups of computing requests from the same source are abstractly parsed into indicator computing logic sets, dimension aggregation combinations, and filter combinations, and the indicator computing logic sets, dimension aggregation combinations, and filter combinations corresponding to each group of computing requests are encapsulated into a computing combination object; each computing combination object is numbered to generate mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the computing combination object; Traversing each of the mapping structure data, using the calculation rule description of the calculation combination object in the mapping structure data to process the data, and outputting composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result; Each distributed working node performs data aggregation according to the data dimensions in the composite key-value pair structure data, and processes the data using the computational logic of the computational combination object that has a mapping relationship with the number ID in the composite key-value pair structure data, and outputs a data aggregation result, wherein the data aggregation result includes the number ID.

2. The method according to claim 1, characterized in that Before each distributed working node aggregates data according to the data dimension in the composite key-value pair structure data, processes data using the computing logic of the computing combination object that has a mapping relationship with the serial number ID in the composite key-value pair structure data, and outputs the data aggregation result, the method further includes: The local working node performs data pre-aggregation according to the data dimensions in the composite key-value pair structure data, and processes local data using the computational logic of the computational combination object that has a mapping relationship with the serial number ID in the composite key-value pair structure data, and outputs a data pre-aggregation result, wherein the data pre-aggregation result includes the serial number ID.

3. The method according to claim 1 or 2, characterized in that: The calculation logic includes a general expression calculation library and a custom calculation function, wherein the custom calculation function completes object instantiation and high-performance calling through reflection.

4. The method according to claim 2, characterized in that: Also includes: The Roaring BitMap algorithm is introduced to integrate deduplication statistics into the bitmap data results, and data pre-aggregation is completed by performing sum and difference calculations on the bitmap.

5. The method according to claim 1, characterized in that Also includes: During the program startup phase, the mapping structure data is stored in global variables and distributed computing contexts, so that the driver program and each working node can obtain the mapping structure data.

6. A multi-dimensional and multi-index data processing device, characterized in that: include: Abstract parsing unit, number mapping unit, traversal data processing unit and data aggregation unit, The abstract parsing unit is used to abstractly parse multiple groups of computing requests of the same source into an indicator computing logic set, a dimension aggregation combination and a filter combination respectively during the program startup phase, and encapsulate the indicator computing logic set, the dimension aggregation combination and the filter combination corresponding to each group of computing requests into a computing combination object respectively; The number mapping unit is used to number each calculation combination object and generate mapping structure data, wherein the mapping structure data contains a mapping relationship between the number ID and the calculation combination object; The traversal data processing unit is used to traverse each of the mapping structure data, use the calculation rule description of the calculation combination object in the mapping structure data to process the data, and output composite key-value pair structure data, wherein the composite key in the composite key-value pair structure data is the number ID and the data dimension, and the value in the composite key-value pair structure data is the indicator data processing result; The data aggregation unit is used to perform data aggregation according to the data dimensions in the composite key-value pair structure data at each distributed working node, and use the computing logic of the computing combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process the data, and output a data aggregation result, wherein the data aggregation result includes the number ID.

7. The device according to claim 6, characterized in that Also includes: Data pre-aggregation unit, A data pre-aggregation unit is used to perform data pre-aggregation at a local working node according to the data dimension in the composite key-value pair structure data, and use the calculation logic of the calculation combination object that has a mapping relationship with the number ID in the composite key-value pair structure data to process local data, and output a data pre-aggregation result, wherein the data pre-aggregation result includes the number ID.

8. The device according to claim 6, characterized in that Also includes: Data storage unit, The data storage unit is used to store the mapping structure data in global variables and distributed computing contexts during the program startup phase, so that the driver program and each working node can obtain the mapping structure data.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the multi-dimensional and multi-index data processing method according to any one of claims 1 to 5 is implemented.

10. An electronic device, characterized in that: The electronic device includes at least one processor, and at least one memory and a bus connected to the processor; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the multi-dimensional and multi-indicator data processing method as described in any one of claims 1 to 5.

Images