A data analysis method and device, electronic equipment and computer storage medium

Abstract

The application provides a data analysis method and device, electronic equipment and computer storage medium, the method comprises the following steps: receiving a query instruction, determining the index dimension information table corresponding to the query instruction; the complex index in the index dimension information table is split, and the initial split index data of the complex index is obtained, and the number of complex indexes is at least one; the initial split index data is processed to determine the corresponding target split index data; the target split index data is associated with the dimension to obtain the data table corresponding to each complex index; and the target expression is executed based on the data in the data table corresponding to the complex index. In the scheme, the complex index in the index dimension information table is split, and logical analysis, dimension association and other processes are performed, and then the corresponding expression is executed. Through the above manner, the index calculation time can be reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data analysis method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] With the rapid development of the e-commerce industry, online shopping is becoming increasingly popular. Therefore, in order to attract more users and increase stickiness to e-commerce applications, it is necessary for e-commerce procurement, sales, and operations to understand the current user structure and user growth and sales growth of each user segment. This is to quickly identify the strong and weak departments, categories, and brands driving growth, and to help the procurement and sales side understand the current growth trend by comparing the year-on-year and month-on-month data of strong and weak businesses. This will enable the procurement and sales side to formulate more targeted category operation strategies from multiple dimensions.

[0003] Currently, the common approach is to use the MOLAP architecture and calculate metrics across all dimensions offline using the Spark computing engine. However, since the complexity of the metrics is uncertain and the amount of data for the dimensions and metrics is large, the metric calculation task is quite large, resulting in a long calculation time. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a data analysis method, apparatus, electronic device, and computer storage medium to solve the problem of long calculation time for indicators in the prior art.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of this invention discloses a data analysis method applied to a data processing system, the method comprising:

[0007] Receive a query command and determine the indicator dimension information table corresponding to the query command;

[0008] The complex indicators in the indicator dimension information table are split to obtain the initial split indicator data of the complex indicators, and the number of complex indicators is at least one.

[0009] The initial splitting indicator data is processed to determine the corresponding target splitting indicator data;

[0010] The target split indicator data are correlated dimensionally to obtain a data table corresponding to each complex indicator.

[0011] The target expression is executed based on the data in the data table corresponding to the complex indicator, wherein the target expression is the indicator expression corresponding to the complex indicator obtained from the indicator dimension information table.

[0012] Optionally, the step of splitting the complex indicators in the indicator dimension information table to obtain the initial split indicator data of the complex indicators includes:

[0013] Search the structural data corresponding to the complex indicator from the indicator dimension information table;

[0014] The complex indicator is split based on the sub-indicators in the structured data to obtain initial split indicator data corresponding to each sub-indicator.

[0015] Optionally, processing the initial splitting indicator data to determine the corresponding target splitting indicator data includes:

[0016] If the query instruction does not carry a dimension modification request, and it is determined that the data corresponding to the indicator filtering conditions in the initial split indicator data needs to be modified, the data corresponding to the indicator filtering conditions in the initial split indicator data is modified based on the data in the snowflake model and the initial split indicator data.

[0017] If the query instruction carries a dimension modification request, the data corresponding to the indicator grouping dimension in the initial split indicator data is modified based on the dimension modification request;

[0018] Use the modified initial split metric data as the target split metric data.

[0019] Optionally, if it is determined that the data corresponding to the indicator filtering conditions in the split indicator data needs to be modified, based on the data in the snowflake model and the initial split indicator data, the data corresponding to the indicator filtering conditions in the initial split indicator data is modified, including:

[0020] Determine whether the data corresponding to the indicator filtering conditions needs to be modified based on the meaning of the sub-indicators in the initial split indicator data;

[0021] If necessary, the data corresponding to the index filtering conditions can be modified based on the data in the snowflake model.

[0022] Optionally, the target split indicator data can be dimensionally correlated to obtain a data table corresponding to each complex indicator, including:

[0023] Use the statements corresponding to the target split index data to query the data in the snowflake model and obtain the corresponding result data;

[0024] The result data is correlated dimensionally according to the indicator dimension data to obtain a data table corresponding to each complex indicator. The indicator dimension data is determined from the indicator grouping dimension in the target split indicator data.

[0025] A second aspect of the present invention discloses a data processing apparatus for use in a data processing system, the apparatus comprising:

[0026] The acquisition unit is used to receive a query instruction and determine the indicator dimension information table corresponding to the query instruction;

[0027] A splitting unit is used to split the complex indicators in the indicator dimension information table to obtain the initial split indicator data of the complex indicators, wherein the number of complex indicators is at least one.

[0028] The logic parsing unit is used to process the initial splitting indicator data and determine the corresponding target splitting indicator data;

[0029] The dimension association unit is used to perform dimension association on the target split indicator data to obtain a data table corresponding to each complex indicator.

[0030] The processing unit is configured to execute a target expression based on data in the data table corresponding to the complex indicator, wherein the target expression is an indicator expression corresponding to the complex indicator obtained from the indicator dimension information table.

[0031] Optionally, the splitting unit is specifically used to: search for structural data corresponding to the complex indicator from the indicator dimension information table; and split the complex indicator based on the sub-indicators in the structural data to obtain initial split indicator data corresponding to each sub-indicator.

[0032] Optionally, the logic parsing unit is specifically used for: if the query instruction does not carry a dimension modification request, and it is determined that the data corresponding to the indicator filtering conditions in the initial split indicator data needs to be modified, modifying the data corresponding to the indicator filtering conditions in the initial split indicator data based on the data in the snowflake model and the initial split indicator data; if the query instruction carries a dimension modification request, modifying the data corresponding to the indicator grouping dimensions in the initial split indicator data based on the dimension modification request; and using the modified initial split indicator data as the target split indicator data.

[0033] A third aspect of the present invention discloses an electronic device for running a program, wherein the program executes a data analysis method as shown in the first aspect of the present invention.

[0034] A fourth aspect of the present invention discloses a computer storage medium comprising a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform a data analysis method as shown in the first aspect of the present invention.

[0035] Based on the above embodiments of the present invention, a data analysis method, apparatus, electronic device, and computer storage medium are provided. The method includes: receiving a query instruction and determining an indicator dimension information table corresponding to the query instruction; splitting complex indicators in the indicator dimension information table to obtain initial split indicator data for the complex indicators, wherein the number of complex indicators is at least one; processing the initial split indicator data to determine corresponding target split indicator data; performing dimension association on the target split indicator data to obtain a data table corresponding to each complex indicator; and executing a target expression based on the data in the data table corresponding to the complex indicator, wherein the target expression is an indicator expression corresponding to the complex indicator obtained from the indicator dimension information table. In the embodiments of the present invention, by splitting complex indicators in the indicator dimension information table, performing logical parsing, dimension association, and other processing, and then executing the corresponding expression, the indicator calculation time can be reduced. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the architecture of a data analysis system according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the snowflake module according to an embodiment of the present invention;

[0039] Figure 3 This is a flowchart illustrating a data analysis method according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram illustrating the index splitting process in an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram illustrating the generation of a data table based on dimensional association, as shown in an embodiment of the present invention.

[0042] Figure 6 This is a schematic diagram illustrating the determination of target splitting index data in an embodiment of the present invention;

[0043] Figure 7 This is a schematic diagram illustrating the logical parsing and determination of target splitting index data in an embodiment of the present invention;

[0044] Figure 8This is a schematic diagram of the structure of a data analysis device according to an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0048] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] See Figure 1This is a schematic diagram of the architecture of a data analysis system according to an embodiment of the present invention. The system includes: a service layer 10, a query engine module 20, a ClickHouse storage module 30, and a HIVE data warehouse 40.

[0050] Service layer 10 includes sub-modules such as application programming interface (API), user permission management, and cache management, which are used to provide users with predefined interfaces.

[0051] The query engine module 20 includes sub-modules for indicator splitting, logical parsing, dimension association, and rule calculation, which are used to execute the engine's functions for indicator transformation and querying.

[0052] The ClickHouse storage module 30, an open-source columnar storage database, is used to store detailed data such as orders and customer profiles. It includes tables such as order tables and new / returning customer tables.

[0053] The HIVE data warehouse 40 includes tables such as order tables, new and returning customer tables, customer profile tables, first-purchase source tables, and platform tables, which are used to store offline data for various themes.

[0054] It should be noted that the data between ClickHouse storage module 30 and HIVE data warehouse 40 is synchronized in real time.

[0055] The sheer volume of data leads to excessively long query execution times, impacting user experience. Large-scale e-commerce platforms handle nearly 10 billion orders annually, with over 100 million users each. Furthermore, the rapid growth of e-commerce users necessitates frequent analysis of order-related data. Therefore, the data in ClickHouse storage module 30 and HIVE data warehouse 40 is stored in a snowflake model to facilitate rapid data retrieval.

[0056] Specifically, ClickHouse storage module 30 and HIVE data warehouse 40 need to store hundreds of millions of data points daily. Since the data volume directly affects query efficiency, it is necessary to perform join operations between data tables and multiple tables to reduce query efficiency. The specific process of building the snowflake model is as follows: join operations are performed on each table according to the data that are related in each table to form a snowflake model for e-commerce user growth data, thereby reducing data redundancy and improving query efficiency.

[0057] For example: Figure 2 As shown, the ClickHouse storage module 30 and the HIVE data warehouse 40 can use the order details table as the fact table and the user profile as the detail table. The two use the department code and category code as FK foreign keys. Based on the FK foreign keys, the order details table is associated with the user profile. Similarly, the order details table is associated with the department information dimension table, the order channel source table, and the new customer tag dimension table through different foreign keys FK.

[0058] See next Figure 2 To achieve better compression efficiency in ClickHouse, the tag names in the original profile table were made into a separate table, resulting in two tables: the user profile table and the tag information dimension table. These two tables are then connected via foreign keys.

[0059] Optionally, the user profile table and the order details table can be JOINed in ClickHouse to return aggregated query results. At this point, the result data will typically contain only a few rows. Finally, the server will add tag names to the result data.

[0060] Based on the architecture shown in the above embodiments of the present invention, a data analysis process is specifically illustrated, including:

[0061] The query engine module 20 receives a query command and obtains the indicator dimension information table corresponding to the query command in the application system;

[0062] Among them, the application system can be an e-commerce application system.

[0063] The indicator splitting submodule in the query engine module 20 splits the complex indicators in the indicator dimension information table to obtain the initial split indicator data of the complex indicators, and the number of complex indicators is at least one.

[0064] The logic parsing submodule in the query engine module 20 processes the initial split indicator data to determine the corresponding target split indicator data.

[0065] The dimension association submodule in the query engine module 20 performs dimension association on the target split indicator data to obtain a data table corresponding to each complex indicator.

[0066] The rule calculation submodule in the query engine module 20 executes the target expression based on the data in the data table corresponding to the complex indicator and feeds back the execution result to the user. The target expression is the indicator expression corresponding to the complex indicator obtained from the indicator dimension information table.

[0067] In this embodiment of the invention, a query command is received, and an indicator dimension information table corresponding to the query command is determined. Complex indicators in the indicator dimension information table are split to obtain initial split indicator data for the complex indicators, with at least one complex indicator. The initial split indicator data is processed to determine the corresponding target split indicator data. The target split indicator data is dimensionally associated to obtain a data table corresponding to each complex indicator. A target expression is executed based on the data in the data table corresponding to the complex indicator. This solution reduces indicator calculation time by splitting complex indicators in the indicator dimension information table, performing logical parsing, and dimension association. Furthermore, using a snowflake model to store data allows the queried data to be found quickly, thereby reducing query time.

[0068] See Figure 3 The following is a flowchart illustrating a data analysis method according to an embodiment of the present invention. The method includes:

[0069] Step S301: Receive a query instruction and determine the indicator dimension information table corresponding to the query instruction.

[0070] In step S301, the query instruction may carry a dimension modification request.

[0071] Optionally, users, i.e., purchasing, sales, or operations personnel, can input query commands through the query engine based on the user data they want to understand.

[0072] In the specific implementation step S301, a query instruction is received, and the indicator dimension information table corresponding to the query instruction in the application system is obtained through the query request.

[0073] It should be noted that the indicator dimension information table includes a lot of structured data, which can include indicator codes, indicator names, expressions, and sub-indicators.

[0074] For example: Table (1) shows the indicator dimension information table. The indicator codes in Table (1) include indicator codes userCnt, userCntLastYear, userCntLastPeriod, userCntYOY and userCntMOM. The indicator code userCnt corresponds to the number of users, and the corresponding expression and sub-indicator are empty. The indicator code userCntLastYear corresponds to the number of users last year, and the corresponding expression and sub-indicator are empty. The indicator code userCntLastPeriod corresponds to the number of users in the previous period, and the corresponding expression and sub-indicator are empty. The indicator code userCntYOY corresponds to the year-on-year change in the number of users, and the corresponding expression is userCnt / userCntLastYear-1, and the corresponding sub-indicator is userCnt,userCntLastYear. The indicator code userCntMOM corresponds to the month-on-month change in the number of users, and the corresponding expression is userCnt / userCntLastPeriod-1, and the corresponding sub-indicator is userCnt,userCntLastPeriod.

[0075] Table (1):

[0076]

[0077] Step S302: The complex indicators in the indicator dimension information table are split to obtain the initial split indicator data of the complex indicators.

[0078] In the specific implementation step S302, the number of the complex indicators is at least one.

[0079] In this embodiment of the invention, the indicators with empty expressions in the indicator dimension information table are basic indicators, while the rest are composite indicators. Composite indicators are obtained by calculating the basic indicators based on their expressions.

[0080] It should be noted that the specific implementation process of step S302 includes the following steps:

[0081] Step S11: Find the structural data corresponding to the complex indicator from the indicator dimension information table.

[0082] In the specific implementation step S11, the indicator code containing the expression in the indicator dimension information table is searched, and the indicator name, expression and sub-indicator corresponding to the indicator code are determined.

[0083] Step S12: Based on the sub-indicators in the structured data, the complex indicator is split to obtain the initial split indicator data corresponding to each sub-indicator.

[0084] In the specific implementation step S12, the complex indicator is split based on the data of the sub-indicators to obtain the initial split indicator data corresponding to each sub-indicator.

[0085] Optionally, if there are multiple complex indicators, and the initial split indicator data of the sub-indicators are the same after splitting, they can be merged.

[0086] For example: Search the indicator dimension information table, i.e., the indicator codes userCntYOY and userCntMOM in table (1) that contain expressions, and determine that the indicator name of the indicator code userCntYOY is year-on-year change in number of users, the corresponding expression is userCnt / userCntLastYear-1, and the corresponding sub-indicator is userCnt,userCntLastYear; the indicator code userCntMOM corresponds to the indicator name of month-on-month change in number of users, the corresponding expression is userCnt / userCntLastPeriod-1, and the corresponding sub-indicator is userCnt,userCntLastPeriod.

[0087] For the complex metric userCntYOY, it is split into its sub-metrics userCnt and userCntLastYear, yielding initial split metric data for each sub-metric. Similarly, for the complex metric userCntMOM, it is split into its sub-metrics userCnt and userCntLastPeriod, yielding initial split metric data for each sub-metric. Since the initial split metric data for the two sub-metrics userCnt are identical, they are merged. Figure 4 As shown.

[0088] The initial split metric data for the sub-metric userCnt includes the metric name userCnt, the metric grouping dimensions quarter and deptId1, and the metric filtering conditions filters for the time field from 2020-11-01 to 2020-11-07; the initial split metric data for the sub-metric userCntLastYear includes the metric name userCntLastYear, the metric grouping dimensions quarter and deptId1, and the metric filtering conditions filters for the time field from 2020-11-01 to 2020-11-07; the initial split metric data for the sub-metric userCntLastPeriod includes the metric name userCntLastPeriod, the metric grouping dimensions quarter and deptId1, and the metric filtering conditions filters for the time field from 2020-11-01 to 2020-11-07.

[0089] Step S303: Process the initial splitting index data to determine the corresponding target splitting index data.

[0090] In the specific implementation of step S303, the initial split index data is subjected to logical parsing functions such as drill-down, roll-up, and time data analysis to modify the initial split index data and determine the corresponding target split index data.

[0091] Step S304: Perform dimensional correlation on the target split indicator data to obtain a data table corresponding to each complex indicator.

[0092] It should be noted that the specific implementation process of step S304 includes the following steps:

[0093] Step S21: Use the statement corresponding to the target split index data to query the data in the snowflake model and obtain the corresponding result data.

[0094] In the specific implementation step S21, firstly, the target split indicator data is used to generate a database query statement SQL through MyBatis or other ORM frameworks, and the Snowflake model is queried to obtain the corresponding returned result data.

[0095] It should be noted that the SQL returned data is registered as a Table object for reading and writing chart data using Tablesaw.

[0096] Step S22: Perform dimensional correlation on the result data according to the target indicator dimension to obtain the data table corresponding to each complex indicator.

[0097] In step S22, the target indicator dimension is determined from the indicator grouping dimension in the target split indicator data.

[0098] In the specific implementation step S22, any one of the indicator dimensions in the indicator grouping dimension of the target split indicator data is specified as the target indicator dimension, and the result data of each sub-indicator is dimensionally associated according to the target indicator dimension to obtain the data table corresponding to each complex indicator.

[0099] For example: Suppose that if the target breakdown data for the sub-metric `userCnt` has `quarter` as 1 and `deptId1` as 123, the corresponding result data is 10; if `quarter` is 1 and `deptId1` is 234, the corresponding result data is 20; if `quarter` is 1 and `deptId1` is 345, the corresponding result data is 30. If the target breakdown data for the sub-metric `userCntLastYear` has `quarter` as 1 and `deptId1` as 123, the corresponding result data is 1; if `quarter` is 1 and `deptId1` is 234, the corresponding result data is 2. If the target breakdown data for the sub-metric `userCntLastPeriod` has `quarter` as 1 and `deptId1` as 123, the corresponding result data is 5; if `quarter` is 1 and `deptId1` is 234, the corresponding result data is 6. Using `deptId1` as the target metric dimension, and performing dimension association based on the result data of each sub-metric with `deptId1`, we obtain the data table corresponding to each complex metric, such as... Figure 5 As shown.

[0100] Step S305: Execute the target expression based on the data in the data table corresponding to the complex indicator.

[0101] In step S305, the target expression is the indicator expression corresponding to the complex indicator obtained from the indicator dimension information table.

[0102] In the specific implementation step S305, the indicator expression corresponding to the complex indicator is obtained from the indicator dimension information table, and the data columns in the data table are subjected to arithmetic operations to generate the indicator. Based on the indicator expression, the column names to be calculated are specified, and the specified columns are subjected to arithmetic operations. That is, the Table object in Tablesaw is used as an input parameter, and the expression requesting the indicator is specified to complete the rule calculation.

[0103] Optionally, it also includes: feeding back the execution result obtained from executing the target expression to the user.

[0104] Optionally, the target expression can also be input by the user based on actual situation, experience, or multiple experiments.

[0105] In this embodiment of the invention, a query command is received, and an indicator dimension information table corresponding to the query command is determined. Complex indicators in the indicator dimension information table are split to obtain initial split indicator data for the complex indicators, with at least one complex indicator. The initial split indicator data is processed to determine the corresponding target split indicator data. The target split indicator data is dimensionally associated to obtain a data table corresponding to each complex indicator. A target expression is executed based on the data in the data table corresponding to the complex indicator. This solution reduces indicator calculation time by splitting complex indicators in the indicator dimension information table, performing logical parsing, and dimension association. Furthermore, using a snowflake model to store data allows the queried data to be found quickly, thereby reducing query time.

[0106] Based on the data analysis method shown above, the specific implementation step S303, which processes the initial splitting indicator data to determine the corresponding target splitting indicator data, includes the following steps:

[0107] Step S31: Determine whether the query instruction carries a dimension modification request. If yes, proceed to step S32; otherwise, proceed to step S33.

[0108] In the specific implementation of step S31, the query request is input to the logic parsing module. The logic parsing module determines whether it is a dimension modification request. If yes, step S32 is executed; otherwise, step S33 is executed.

[0109] Step S32: Modify the data corresponding to the indicator grouping dimension in the initial split indicator data based on the dimension modification request.

[0110] In the specific implementation of step S32, based on the dimension modification request, it is determined whether drill-down parsing of the indicator grouping dimension is required. If so, drill-down parsing of the indicator grouping dimension is performed based on the data in the snowflake model to determine the dimension data that needs to be modified, and the data corresponding to the indicator grouping dimension is modified based on the dimension data. If not, based on the dimension modification request, it is determined whether roll-up parsing of the indicator grouping dimension is required. If so, roll-up parsing of the indicator grouping dimension is performed based on the data in the snowflake model to determine the dimension data that needs to be modified, and the data corresponding to the indicator grouping dimension is modified based on the dimension data. Then, step S33 is executed.

[0111] It should be noted that, since there are dimension modification requests, there will inevitably be situations where drill-down parsing or roll-up parsing is performed on the dimension modification requests.

[0112] For example, the indicator grouping dimension stores the dimension data of these two levels of departments. The dimension modification request indicates that the indicator data corresponding to the third level department should be viewed. At this time, it can be determined that the indicator grouping dimension needs to be drilled down. Based on the data in the snowflake model, the indicator grouping dimension is drilled down to determine the dimension data corresponding to the third level department in the snowflake model as the dimension data to be modified. Based on the dimension data, the data corresponding to the indicator grouping dimension is modified. The so-called roll-up parsing is the reverse.

[0113] Step S33: Determine whether it is necessary to modify the data corresponding to the indicator filtering conditions in the initial split indicator data. If it is determined that it is necessary to modify the data corresponding to the indicator filtering conditions in the initial split indicator data, proceed to step S34. If it is determined that it is not necessary to modify the data corresponding to the indicator filtering conditions in the initial split indicator data, determine that it is not necessary to modify the indicator grouping dimensions and indicator filtering conditions filters, and directly proceed to step S35.

[0114] It should be noted that the specific implementation process of step S33 includes the following steps:

[0115] Step S41: Determine whether the data corresponding to the indicator filtering conditions needs to be modified based on the meaning of the sub-indicators in the initial split indicator data. If so, proceed to step S42; otherwise, do not modify the data corresponding to the indicator filtering conditions.

[0116] In the specific implementation of step S41, since the time period in the split indicator filtering condition is the current time when the query command is executed, it is necessary to determine whether the meaning of the sub-indicator contains words that represent the past or future. If it does, step S42 is executed; if it does not, the data corresponding to the indicator filtering condition is not modified.

[0117] Step S42: Modify the data corresponding to the index filtering conditions based on the data in the snowflake model.

[0118] In the specific implementation step S42, the time tension of the indicator filtering condition is analyzed based on the data in the snowflake model, so as to find the data in the snowflake model through the meaning of the sub-indicator, determine the filtering condition data that needs to be modified, and modify the data corresponding to the indicator filtering condition based on the filtering condition data.

[0119] For example: Based on the data in the snowflake model, the time frame of the indicator filtering condition is analyzed. Using the sub-indicator `userCntLastYear` (meaning the number of users last year), the data in the snowflake model is searched for the time field of the filters, from 2019-11-01 to 2019-11-07. The time field of the filters in the sub-indicator `userCntLastYear` is modified from 2020-11-01 to 2020-11-07 to 2019-11-01 to 2019-11-07. Figure 6 As shown.

[0120] It should be noted that the execution order of steps S32 and S33 can be as shown above, or step S33 can be executed first and then step S32.

[0121] Accordingly, the method shown in steps S32 and S33 above can be used... Figure 7 The schematic diagram of the architecture shown is processed.

[0122] Figure 7 It also includes slice parsing, which means that in addition to the time parsing, drill-down parsing, and roll-up parsing shown above, technicians can also set other parsing methods.

[0123] It should be noted that other slice parsing functions can also be set to modify other data in the initial split indicator data.

[0124] Step S34: Use the modified initial split metric data as the target split metric data.

[0125] Step S35: Use the initial split indicator data as the target split indicator data.

[0126] In this invention, the initial splitting indicator data is processed to determine the corresponding target splitting indicator data; then, after processing such as dimension association, the corresponding expression is executed. This method reduces indicator calculation time. Furthermore, using a snowflake model to store data allows for quick retrieval of query data, thereby reducing query time.

[0127] Based on the data analysis method shown in the above embodiments of the present invention, the present invention also shows a corresponding data analysis device, which can be installed in a data analysis system, such as... Figure 8 The diagram shown is a structural schematic of a data analysis device according to an embodiment of the present invention. The device includes:

[0128] The acquisition unit 801 is used to receive a query instruction and determine the indicator dimension information table corresponding to the query instruction.

[0129] The splitting unit 802 is used to split the complex indicators in the indicator dimension information table to obtain the initial split indicator data of the complex indicators, wherein the number of complex indicators is at least one.

[0130] The logic parsing unit 803 is used to process the initial splitting indicator data and determine the corresponding target splitting indicator data;

[0131] The dimension association unit 804 is used to perform dimension association on the target split indicator data to obtain a data table corresponding to each complex indicator.

[0132] The processing unit 805 is used to execute a target expression based on the data in the data table corresponding to the complex indicator, wherein the target expression is the indicator expression corresponding to the complex indicator obtained from the indicator dimension information table.

[0133] It should be noted that the specific principles and execution processes of each unit in the data analysis device disclosed in the above embodiments of the present invention are the same as those of the data analysis method shown in the above embodiments of the present invention. Please refer to the corresponding parts of the data analysis method disclosed in the above embodiments of the present invention, which will not be repeated here.

[0134] In this embodiment of the invention, a query command is received, and an indicator dimension information table corresponding to the query command is determined. Complex indicators in the indicator dimension information table are split to obtain initial split indicator data for the complex indicators, with at least one complex indicator. The initial split indicator data is processed to determine the corresponding target split indicator data. The target split indicator data is dimensionally associated to obtain a data table corresponding to each complex indicator. A target expression is executed based on the data in the data table corresponding to the complex indicator. This solution reduces indicator calculation time by splitting complex indicators in the indicator dimension information table, performing logical parsing, and dimension association. Furthermore, using a snowflake model to store data allows the queried data to be found quickly, thereby reducing query time.

[0135] Optionally, based on the data analysis apparatus shown in the above embodiments of the present invention, the splitting unit 802 is specifically used for: searching for structural data corresponding to the complex indicator from the indicator dimension information table; splitting the complex indicator based on the sub-indicators in the structural data to obtain initial split indicator data corresponding to each sub-indicator.

[0136] Optionally, based on the data analysis apparatus shown in the above embodiments of the present invention, the logic parsing unit 803 is specifically used for: if the query instruction does not carry a dimension modification request, and it is determined that the data corresponding to the indicator filtering conditions in the initial split indicator data needs to be modified, modifying the data corresponding to the indicator filtering conditions in the initial split indicator data based on the data in the snowflake model and the initial split indicator data; if the query instruction carries a dimension modification request, modifying the data corresponding to the indicator grouping dimensions in the initial split indicator data based on the dimension modification request; and using the modified initial split indicator data as the target split indicator data.

[0137] Optionally, based on the data analysis apparatus shown in the above embodiments of the present invention, if it is determined that the data corresponding to the indicator filtering conditions in the split indicator data needs to be modified, the logic parsing unit 803, based on the data in the snowflake model and the initial split indicator data, modifies the data corresponding to the indicator filtering conditions in the initial split indicator data. Specifically, it is used for:

[0138] Determine whether the data corresponding to the indicator filtering conditions needs to be modified based on the meaning of the sub-indicators in the initial split indicator data;

[0139] If necessary, the data corresponding to the index filtering conditions can be modified based on the data in the snowflake model.

[0140] Optionally, based on the data analysis apparatus shown in the above embodiments of the present invention, the dimension association unit 804 is specifically used for:

[0141] Use the statements corresponding to the target split index data to query the data in the snowflake model and obtain the corresponding result data;

[0142] The result data is correlated dimensionally according to the indicator dimension data to obtain a data table corresponding to each complex indicator. The indicator dimension data is determined from the indicator grouping dimension in the target split indicator data.

[0143] This invention also discloses an electronic device for running database stored procedures, wherein running the database stored procedures involves executing the above-described... Figure 2 Publicly available data processing methods.

[0144] This invention also discloses a computer storage medium, the storage medium including a stored database procedure, wherein, during the execution of the stored database procedure, the device where the storage medium is located is controlled to execute the above-mentioned... Figure 2 Publicly available data processing methods.

[0145] In the context of this disclosure, a computer storage medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data analysis method, characterized in that, Applied to a data processing system, the method includes: Receive a query command and determine the indicator dimension information table corresponding to the query command; The structural data corresponding to the complex indicator is retrieved from the indicator dimension information table; wherein, the indicator dimension information table includes multiple structural data, and the multiple structural data include indicator code, indicator name, expression and sub-indicator; The complex indicator is split based on the sub-indicators in the structured data to obtain initial split indicator data corresponding to each sub-indicator; wherein, the complex indicator is obtained by calculating the basic indicator according to the expression. If there are multiple complex indicators, merge the initial split indicator data with the same sub-indicators after splitting; The initial splitting indicator data is processed to determine the corresponding target splitting indicator data; The target split indicator data are correlated dimensionally to obtain a data table corresponding to each complex indicator. The target expression is executed based on the data in the data table corresponding to the complex indicator, wherein the target expression is the indicator expression corresponding to the complex indicator obtained from the indicator dimension information table; The process of processing the initial splitting indicator data to determine the corresponding target splitting indicator data includes: If the query instruction carries a dimension modification request, determine whether drill-down parsing of the indicator grouping dimension is required based on the dimension modification request. If required, drill-down parsing of the indicator grouping dimension is performed based on the data in the snowflake model to determine the dimension data; the data corresponding to the indicator grouping dimension is modified based on the dimension data. If not required, determine whether roll-up parsing of the indicator grouping dimension is required based on the dimension modification request. If required, roll-up parsing of the indicator grouping dimension is performed based on the data in the snowflake model to determine the dimension data, and the data corresponding to the indicator grouping dimension is modified based on the dimension data. Use the modified initial split metric data as the target split metric data.

2. The method according to claim 1, characterized in that, If it is determined that the data corresponding to the indicator filtering conditions in the split indicator data needs to be modified, based on the data in the snowflake model and the initial split indicator data, the data corresponding to the indicator filtering conditions in the initial split indicator data is modified, including: Determine whether the data corresponding to the indicator filtering conditions needs to be modified based on the meaning of the sub-indicators in the initial split indicator data; If necessary, the data corresponding to the index filtering conditions can be modified based on the data in the snowflake model.

3. The method according to claim 1, characterized in that, The target segmentation index data is correlated dimensionally to obtain a data table corresponding to each complex index, including: Use the statements corresponding to the target split index data to query the data in the snowflake model and obtain the corresponding result data; The result data is correlated dimensionally according to the indicator dimension data to obtain a data table corresponding to each complex indicator. The indicator dimension data is determined from the indicator grouping dimension in the target split indicator data.

4. A data processing apparatus, characterized in that, The device, used in a data processing system, includes: The acquisition unit is used to receive a query instruction and determine the indicator dimension information table corresponding to the query instruction; A splitting unit is used to find the structural data corresponding to the complex indicator from the indicator dimension information table; split the complex indicator based on the sub-indicators in the structural data to obtain initial split indicator data corresponding to each sub-indicator; if there are multiple complex indicators, the initial split indicator data with the same sub-indicators after splitting are merged; wherein, the indicator dimension information table includes multiple structural data, the multiple structural data including indicator code, indicator name, expression and sub-indicator; the complex indicator is obtained by calculating the expression based on the basic indicator; The logic parsing unit is used to process the initial splitting indicator data and determine the corresponding target splitting indicator data; The dimension association unit is used to perform dimension association on the target split indicator data to obtain a data table corresponding to each complex indicator. The processing unit is configured to execute a target expression based on the data in the data table corresponding to the complex indicator, wherein the target expression is an indicator expression corresponding to the complex indicator obtained from the indicator dimension information table. The logic parsing unit is specifically used for: The initial splitting metric data is processed to determine the corresponding target splitting metric data, including: If the query instruction carries a dimension modification request, determine whether drill-down parsing of the indicator grouping dimension is needed based on the dimension modification request. If so, drill-down parsing of the indicator grouping dimension is performed based on the data in the snowflake model to determine the dimension data; the data corresponding to the indicator grouping dimension is modified based on the dimension data. If not, determine whether roll-up parsing of the indicator grouping dimension is needed based on the dimension modification request. If so, roll-up parsing of the indicator grouping dimension is performed based on the data in the snowflake model to determine the dimension data, and the data corresponding to the indicator grouping dimension is modified based on the dimension data; the modified initial split indicator data is used as the target split indicator data.

5. An electronic device, characterized in that, The electronic device is used to run a program, wherein the program executes the data analysis method as described in any one of claims 1-3.

6. A computer storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the data analysis method as described in any one of claims 1-3.

Images