Multidimensional data analysis method, data analysis device and data analysis system

Abstract

The present application provides a data analysis method, data analysis device, and data analysis system for multidimensional data. The method includes: obtaining a target data table, analyzing the target data table to obtain a fact table and multiple dimension tables; analyzing the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields; analyzing each dimension field to determine a first association type between each dimension field and combining the dimension fields according to the first association type to obtain multiple first dimension groups; adding the metric fields corresponding to the dimension fields in each first dimension group to the corresponding first dimension group to obtain multiple second dimension groups, and constructing corresponding data models based on each second dimension group to obtain multiple cubes; and performing data analysis on each cube to obtain multiple data analysis results. This method solves the problem in the prior art of excessive computational complexity and excessive resource consumption caused by combining all dimensions when performing data analysis on a data model.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data analysis method, a data analysis device, a computer-readable storage medium, and a data analysis system for multidimensional data. Background Art

[0002] Online Analytical Processing (OLAP) systems are the primary application of data warehouse systems, specifically designed to support complex analytical operations, with a focus on supporting decision-makers and senior management. OLAP systems can be categorized by their data storage format: relational OLAP (ROLAP), multidimensional OLAP (MOLAP), and hybrid OLAP (HOLAP). MOLAP physically stores the multidimensional data used for OLAP analysis as multidimensional arrays, forming a "cube" structure. MOLAP operations include drill-down, roll-up, slice, dice, and pivot.

[0003] MOLAP multidimensional analysis based on massive data sets first requires computational complexity. Second, there's the issue of dimensions. Multidimensional analysis often requires analysis across more than a dozen dimensions, and these dimensions must be combined for analysis, resulting in a dramatic increase in the amount of computation required. Third, there's the inability to guarantee the timeliness of multidimensional analysis query results.

[0004] Existing technologies only consider how to build data models and set dimensions and metrics, but fail to consider the impact of the number of dimensions on subsequent analysis. Existing technologies pre-calculate and combine each dimension, storing the results in a database or distributed storage. Assuming there are four dimensions, there are only 2^4 = 16 possible dimension combinations. However, in reality, the number of dimensions in multidimensional analysis is typically far greater than four. For example, if there are 20 dimensions, there are 2^20 = 1,048,576 possible dimension combinations. This magnitude of dimensional combinations places enormous pressure on data computation and storage. Summary of the Invention

[0005] The main purpose of this application is to provide a data analysis method, a data analysis device, a computer-readable storage medium and a data analysis system for multidimensional data, so as to at least solve the problem in the prior art that when performing data analysis on a data model, all dimensions are combined, resulting in excessive computational complexity and excessive resource consumption.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data analysis method for multidimensional data is provided, comprising: obtaining a target data table, and analyzing the target data table to obtain a fact table and multiple dimension tables, wherein the target data table is a data table to be analyzed, the fact table is used to store event records, and the dimension table is used to store the characteristics of the event records in the fact table; according to the analysis of the fact table and each dimension table, multiple dimension fields and multiple measurement fields are obtained, wherein the dimension field is a field for describing the characteristics of the event record, and the measurement field is a field for the numerical value in the event record, and one dimension field corresponds to one measurement field; The dimension fields are analyzed to determine the first association type between each of the dimension fields and the dimension fields are combined according to the first association type to obtain multiple first dimension groups, wherein the first association type includes similar fields, mutually exclusive fields and unrelated fields, and the first association type between any two dimension fields in the first dimension group is not the similar field or the mutually exclusive field; the metric fields corresponding to the dimension fields in each of the first dimension groups are added to the corresponding first dimension group to obtain multiple second dimension groups, and corresponding data models are constructed according to each of the second dimension groups to obtain multiple cubes; data analysis is performed on each cube to obtain multiple data analysis results.

[0007] Optionally, multiple dimension fields and multiple metric fields are obtained based on the analysis of the fact table and each dimension table, including: comparing the fields in the fact table and each dimension table to obtain multiple associated fields and meaningless fields, the associated fields are the same fields in the dimension table and the fact table, and the meaningless fields are the fields in the fact table and the dimension table used to characterize system status control and configuration control; determining the second association type between the dimension table and the fact table based on the proportion of the associated fields in each dimension table, the second association type includes inner association, left outer association, right outer association and full outer association, and the second association type corresponds one-to-one to the dimension table; deleting the meaningless fields in the fact table and each dimension table, and constructing a star data model based on the fact table, each dimension table and each second association type; analyzing each field in the star data model to obtain multiple dimension fields and metric fields.

[0008] Optionally, each dimension field is analyzed to determine the first association type between each dimension field, including: performing semantic analysis on each dimension field, and determining the first association type between the dimension fields whose corresponding semantic similarity exceeds a first threshold as the similar field; consulting a preset information table to obtain the dimension fields whose corresponding first association type is the mutually exclusive field, the preset information table including the dimension fields that are not combined in the data analysis requirements; and determining the first association type between the dimension fields whose first association type is not the similar field and the mutually exclusive field as an unrelated field.

[0009] Optionally, the dimension fields are combined according to the first association type to obtain multiple first dimension groups, including: analyzing each dimension field and classifying the dimension fields corresponding to the same characteristic into the same level to obtain multiple dimension levels; combining the dimension fields according to the first association type and the dimension level to obtain multiple first dimension groups, and any two dimension fields in the first dimension group are not in the same dimension level.

[0010] Optionally, the metric fields corresponding to the dimension fields in each of the first dimension groups are added to the corresponding first dimension groups to obtain multiple second dimension groups, and corresponding data models are constructed based on each of the second dimension groups to obtain multiple cubes, including: an acquisition step, obtaining multiple target associated fields, the target associated fields are the associated fields that have the same fields as the dimension fields in the target first dimension group, and the target first dimension group is any one of the first dimension groups; a determination step, determining a target metric field based on each of the target associated fields, the target metric field is the metric field that has the same field as the associated field; an integration step, adding multiple target metric fields to the target first dimension group to obtain the second dimension group; repeating the acquisition step, the determination step and the integration step at least once in sequence until all the second dimension groups are obtained; setting corresponding operation functions based on the dimension fields and the metric fields in each of the second dimension groups to obtain multiple metric functions, the metric functions including sum, minimum, maximum, average, count, standard deviation and variance; and constructing a data model based on each of the second dimension groups and the corresponding metric functions to obtain the corresponding cube.

[0011] Optionally, data analysis is performed on each cube to obtain multiple data analysis results, including: a first construction step, constructing a grouping SQL statement based on the dimension field in the target cube, and the target cube is any one of the cubes; a second construction step, constructing a value SQL statement based on the metric function in the target cube; a combination step, combining the grouping SQL statement and the value SQL statement to obtain a corresponding target SQL statement; repeating the first construction step, the second construction step and the combination step at least once in sequence until the target SQL statements corresponding to all the cubes are obtained; and executing each of the target SQL statements in sequence to obtain the data analysis results corresponding to each cube.

[0012] Optionally, after performing data analysis on each cube to obtain multiple data analysis results, the method further includes: sorting each dimension field to obtain a serial number corresponding to each dimension field; setting corresponding identification information for each serial number according to the dimension field in each cube, the identification information being used to characterize whether the dimension field corresponding to the serial number exists in the cube; generating a first row key value according to the identification information, and generating a second row key value according to the data analysis result; combining the first row key value and the second row key value to obtain a target row key value, and storing the target row key value to a corresponding position in the distributed cluster.

[0013] According to another aspect of the present application, a data analysis device for multidimensional data is provided, the device comprising: an acquisition unit for acquiring a target data table and analyzing the target data table to obtain a fact table and multiple dimension tables, the target data table being a data table to be analyzed, the fact table being used to store event records, and the dimension table being used to store characteristics of the event records in the fact table; a first analysis unit for analyzing the fact table and each dimension table to obtain multiple dimension fields and multiple measurement fields, the dimension fields being fields for describing characteristics of the event records, the measurement fields being fields for numerical values ​​in the event records, and one dimension field corresponding to one measurement field; a second analysis unit for Each of the dimension fields is analyzed to determine the first association type between each of the dimension fields and the dimension fields are combined according to the first association type to obtain multiple first dimension groups, wherein the first association type includes similar fields, mutually exclusive fields and unrelated fields, and the first association type between any two of the dimension fields in the first dimension group is not the similar field or the mutually exclusive field; a construction unit is used to add the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension group to obtain multiple second dimension groups, and construct a corresponding data model according to each of the second dimension groups to obtain multiple cubes; a third analysis unit is used to perform data analysis on each cube to obtain multiple data analysis results.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0015] According to another aspect of the present application, a data analysis system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0016] Applying the technical solution of the present application, in the data analysis method of the multidimensional data, first, a target data table is obtained, and the target data table is analyzed to obtain a fact table and multiple dimension tables, the target data table is the data table to be analyzed, the fact table is used to store event records, and the dimension table is used to store the characteristics of the event records in the fact table; then, according to the fact table and each of the dimension tables, multiple dimension fields and multiple measurement fields are obtained, the dimension fields are fields for describing the characteristics of the event records, and the measurement fields are fields for the numerical values ​​in the event records, and one dimension field corresponds to one measurement field; then, for each of the dimension fields An analysis is performed to determine the first association type between each of the above-mentioned dimension fields and the above-mentioned dimension fields are combined according to the above-mentioned first association type to obtain multiple first dimension groups. The above-mentioned first association type includes similar fields, mutually exclusive fields and unrelated fields. The above-mentioned first association type between any two of the above-mentioned dimension fields in the above-mentioned first dimension group is not the above-mentioned similar field or the above-mentioned mutually exclusive field; thereafter, the above-mentioned measurement fields corresponding to the above-mentioned dimension fields in each of the above-mentioned first dimension groups are added to the corresponding above-mentioned first dimension group to obtain multiple second dimension groups, and corresponding data models are constructed according to each of the above-mentioned second dimension groups to obtain multiple cubes; finally, data analysis is performed on each cube to obtain multiple data analysis results. The present application performs a meaning analysis on the dimension fields in the data model, determines that fields with similar functions in data analysis are similar fields, and fields that should not be analyzed at the same time are mutually exclusive fields. When combining dimensions, the dimensions and corresponding measurements of mutually similar fields and mutually exclusive fields are set not to exist in the same combination, which greatly reduces unnecessary dimension combinations and solves the problem in the prior art that when performing data analysis on the data model, all dimensions are combined, resulting in excessive calculation and excessive resource occupation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a data analysis method for multidimensional data provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic diagram of a process for analyzing multidimensional data provided in accordance with an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram of a result of dividing a dimension group provided in an embodiment of the present application is shown;

[0020] Figure 4 An algorithm flow chart of a specific multidimensional data analysis method provided according to an embodiment of the present application is shown;

[0021] Figure 5The figure shows a structural block diagram of a multi-dimensional data analysis device provided according to an embodiment of the present application.

[0022] The above drawings include the following reference numerals:

[0023] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0028] OLAP stands for on-line analytical processing. It is a software technology that enables analysts to quickly, consistently, and interactively analyze information from all perspectives, achieving a deep understanding of the data. OLAP systems can be categorized by their data storage format: relational OLAP (ROLAP), multidimensional OLAP (MOLAP), and hybrid OLAP (HOLAP).

[0029] MOLAP: Physically stores the multidimensional data used in OLAP analysis as a multidimensional array, forming a "cube" structure. Dimensional attribute values ​​are mapped to subscript values ​​or subscript ranges in the multidimensional array, while summary data is stored as values ​​in the array's cells.

[0030] Dimension: Simply put, dimension is the angle from which data is observed, such as time dimension or region dimension.

[0031] Measure: A metric is an aggregated statistical value and is the result of an aggregation operation.

[0032] Cube: All fields in a data table or data model can be divided into dimensions and metrics. Within this model, all dimensions can be combined. For n dimensions, there are 2^n possible combinations. For each dimension combination, metrics can be aggregated to form indicators. Simply put, a cube is a collection of metrics aggregated by dimension.

[0033] Roll-up / Drill-up: Aggregate along one (or more) dimensions and observe the summary data in other dimensions after aggregation. This is equivalent to compressing (aggregating) a cube along a certain dimension.

[0034] Drill-down: Drill down along one (or more) dimensions at a finer granularity level to observe the corresponding data in other dimensions. This is equivalent to stretching a cube along a dimension, resulting in finer granularity. For example, stretching the time dimension from quarter to month.

[0035] As introduced in the background technology, the existing technology only considers how to set dimensions and metrics when constructing data models and analyzing data, and combines all dimensions. Assuming there are 20 dimensions, there are 2^20=1048576 dimension combinations, which will put huge pressure on data calculation and storage. In order to solve the problem in the existing technology that when performing data analysis on data models, all dimensions are combined, resulting in excessive calculation and excessive resource consumption, the embodiments of the present application provide a data analysis method, data analysis device, computer-readable storage medium and data analysis system for multidimensional data.

[0036] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1FIG. 1 is a hardware structure block diagram of a mobile terminal for a multi-dimensional data analysis method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0038] The memory 104 can be used to store computer programs, such as software programs and modules of application software, such as the computer program corresponding to the device information display method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] In this embodiment, a data analysis method for multidimensional data running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 Flowchart of the multidimensional data analysis method according to the embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0041] Step S201: Obtain a target data table and analyze the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table.

[0042] Specifically, according to the analysis requirements of data analysis, all data tables that need to be accessed in the database, namely the above-mentioned target data tables, are determined, and then various data recorded in the data tables are analyzed, extracted and classified, namely the above-mentioned fact tables and the above-mentioned dimension tables are obtained.

[0043] In one embodiment, there is only one fact table in the same star schema, and a dimension table may or may not exist. In this embodiment, multiple dimension tables exist at the same time.

[0044] Step S202: Analyze the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields. The dimension fields are fields used to describe characteristics of the event records, and the metric fields are fields containing numerical values ​​in the event records. One dimension field corresponds to one metric field.

[0045] Specifically, the fields in the fact table and the dimension table are divided into the dimension fields and the metric fields according to the contents contained in the fields.

[0046] In a specific implementation, the dimension fields mentioned above may come from a fact table or a dimension table, and the metric fields mentioned above only come from a fact table.

[0047] Step S203: Analyze each of the dimension fields to determine a first association type between each of the dimension fields, and combine the dimension fields according to the first association type to obtain multiple first dimension groups. The first association type includes similar fields, mutually exclusive fields, and unrelated fields. The first association type between any two of the dimension fields in the first dimension group is neither similar fields nor mutually exclusive fields.

[0048] Specifically, semantic analysis is performed on the above-mentioned dimension fields and the above-mentioned measurement fields to determine similar fields, and based on the data analysis requirements and field semantics, the association types between fields that will not be analyzed together during the data analysis process are determined as mutually exclusive fields, and the association types between the remaining fields are determined as unrelated fields. In the process of dimension combination of fields, there are no two dimension fields in the same dimension combination that are similar or mutually exclusive fields. Figure 3As shown, the dimension fields that are meaningless to each other are combined to obtain a plurality of first dimension groups, and there may be overlap or no intersection between the groups.

[0049] Step S204: adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain multiple second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain multiple cubes;

[0050] Specifically, the metric fields corresponding to the dimension fields in the above-mentioned first dimension combinations are determined, and then they are associated one by one to obtain the above-mentioned second dimension group, and then the corresponding data model is constructed according to the dimension fields and metric fields in the second dimension group to obtain the above-mentioned cube.

[0051] Step S205 , performing data analysis on each cube to obtain multiple data analysis results.

[0052] Specifically, data operations are performed on the metric fields corresponding to each dimension field, including summation, maximum value, minimum value statistics, etc. The above operation results are the data analysis results.

[0053] Through this embodiment, first, a target data table is obtained, and the target data table is analyzed to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed, the fact table is used to store event records, and the dimension table is used to store the characteristics of the event records in the fact table; then, multiple dimension fields and multiple measurement fields are obtained based on the analysis of the fact table and the dimension tables. The dimension fields are fields used to describe the characteristics of the event records, and the measurement fields are fields of numerical values ​​in the event records. One dimension field corresponds to one measurement field; then, each dimension field is analyzed to determine the characteristics of each dimension field. The first association type between the dimension fields and the above-mentioned dimension fields are combined according to the above-mentioned first association type to obtain multiple first dimension groups, the above-mentioned first association type includes similar fields, mutually exclusive fields and unrelated fields, and the above-mentioned first association type between any two of the above-mentioned dimension fields in the above-mentioned first dimension group is not the above-mentioned similar field or the above-mentioned mutually exclusive field; then, the above-mentioned metric fields corresponding to the above-mentioned dimension fields in each of the above-mentioned first dimension groups are added to the corresponding above-mentioned first dimension group to obtain multiple second dimension groups, and the corresponding data model is constructed according to each of the above-mentioned second dimension groups to obtain multiple cubes; finally, data analysis is performed on each cube to obtain multiple data analysis results. The present application analyzes the meaning of the dimension fields in the data model, determines that the fields with similar functions in the data analysis are similar fields, and the fields that should not be analyzed at the same time are mutually exclusive fields. When combining dimensions, the dimensions and corresponding metrics of the mutually similar fields and mutually exclusive fields are set not to exist in the same combination, which greatly reduces unnecessary dimension combinations and solves the problem in the prior art that when performing data analysis on the data model, all dimensions are combined, resulting in excessive calculation and excessive resource occupation.

[0054] In order to sort out the data fields in the target data table, in an optional implementation, step S202 includes:

[0055] Step S2021: Compare the fields in the fact table and each dimension table to obtain a plurality of associated fields and meaningless fields. The associated fields are the same fields in the dimension table as in the fact table. The meaningless fields are the fields in the fact table and the dimension table used to represent system status control and configuration control.

[0056] Specifically, the fields stored in the fact table and the dimension table are compared, and the same fields therein are traversed to determine as associated fields. The fact table and the dimension table are linked through the associated fields, and there is no association between the dimension tables.

[0057] In practice, fact tables and dimension tables should theoretically contain only fields that store fact records and describe their characteristics, namely, dimension fields and metric fields. However, in practice, fact tables and dimension tables often contain many fields that are irrelevant to the business, such as those for system status control and configuration control data change events. These fields are not meaningful for business analysis, so they are classified as meaningless. When building the data model, these meaningless fields are eliminated to reduce the possibility of dimension combinations and computational complexity.

[0058] Step S2022: Determine a second association type between the dimension table and the fact table based on the proportion of the association fields in each dimension table. The second association type includes inner join, left outer join, right outer join, and full outer join. Each second association type corresponds to each dimension table one by one.

[0059] Specifically, the association type between the fact table and the dimension table is determined based on the proportion of the fields to which the associated fields belong in all the fields included in the table. For example, based on the proportion in the fact table, when the proportion in the dimension table is higher than that in the fact table, the left outer join association type is selected, that is, the fields to which the above-mentioned associated fields belong in the fact table are retained, and the remaining fields are eliminated to ensure that when a cube is built based on the dimension table, all metric fields have corresponding dimension fields.

[0060] Step S2023: deleting the meaningless fields in the fact table and each dimension table, and constructing a star data model based on the fact table, each dimension table, and each second association type.

[0061] Specifically, the above meaningless fields are deleted, and fields in the fact table or dimension table are removed according to the above second association type between each dimension table and the fact table, and the modified fact table and dimension table are associated according to the associated fields to obtain the above star data model.

[0062] Step S2024: Analyze each of the above fields in the above star data model to obtain the multiple above dimension fields and the above metric fields.

[0063] Specifically, the fields in the star data model are analyzed and divided into dimension fields and measure fields according to the content described by the fields.

[0064] In order to simplify the amount of calculation in the data analysis process, in an optional embodiment, the above step S203 includes:

[0065] Step S2031: performing semantic analysis on each of the dimension fields, and determining the first association type between the dimension fields whose corresponding semantic similarity exceeds a first threshold as the similar fields;

[0066] Specifically, a semantic analysis is performed on each dimension field from the perspective of business meaning, and two dimension fields with similar semantics are determined to be similar fields, such as region and regional scale. These two fields have similar business meanings. Putting them in the same dimension combination for data analysis will only slightly improve the effect and significantly increase the number of combinations and computational complexity. Therefore, similar fields in this application are not combined when combining dimensions.

[0067] Step S2032: Look up a preset information table to obtain the dimension fields corresponding to the mutually exclusive fields of the first association type, wherein the preset information table includes the dimension fields that are not combined in the data analysis requirement;

[0068] Specifically, there are preset information tables corresponding to different data analysis requirements, which include field combinations that will not improve the data analysis effect after being combined during the data analysis process. By referring to the above information tables, it is possible to determine which dimension fields in the current star data model have association types that are mutually exclusive fields.

[0069] Step S2033: Determine the first association type between the dimension fields whose first association type is not the similar field and the mutually exclusive field as an unrelated field.

[0070] Specifically, after determining that the field corresponding to the above first association type is a similar field or a mutually exclusive field, the above first association type between it and the field is determined to be an unrelated field, that is, a field that can be freely combined in dimensions during data analysis.

[0071] In the above embodiment, there are two fields that are mutually exclusive or similar. These two dimension fields can be combined with the remaining n dimension fields for multidimensional analysis. n+1 possible combinations, but in traditional multidimensional data analysis methods, the first type of association is not divided. n+2 possible combinations, the effect of the method of this application is no different from the existing technology, but when there are three fields that are mutually exclusive or similar, these three dimension fields can be combined with the remaining n-1 dimension fields for multidimensional analysis. n possible combinations, which is 2 fewer than traditional multidimensional data analysis methods. n When considering each possible combination, it can be inferred that the more fields corresponding to the first association type are mutually exclusive fields or similar fields, the more obvious the effect of the method of the present application on reducing the amount of calculation is.

[0072] In order to simplify the amount of calculation in the data analysis process, in an optional embodiment, the above step S203 further includes:

[0073] Step S2034: Analyze each of the dimension fields and classify the dimension fields corresponding to the same characteristic into the same level to obtain multiple dimension levels;

[0074] Specifically, the aforementioned dimensional hierarchy is obtained by analyzing the characteristics of the event records described by the dimension fields and grouping the dimensions used to record the same characteristics into the same level. Each level includes at least one dimension field. For example, the time dimensions year, quarter / month, and day; the regional dimensions province, city, and county; and the organizational dimensions first-level, second-level, and third-level institutions can all be categorized into the same dimensional level.

[0075] Step S2035 : Combining the dimension fields according to the first association type and the dimension level to obtain a plurality of first dimension groups, wherein any two dimension fields in the first dimension groups are not in the same dimension level.

[0076] In the embodiment of the present application, the purpose of setting the dimension hierarchy is to reduce the possibility of dimension combinations, that is, to combine the dimension combinations as a whole, that is, to select only one dimension in the dimension hierarchy for combination.

[0077] In order to establish a data model from a dimensional perspective to facilitate data analysis, in an optional implementation, the above step S204 includes:

[0078] Step S2041, an acquisition step, acquiring a plurality of target associated fields, wherein the target associated fields are associated fields that are identical to the dimension fields in the target first dimension group, and the target first dimension group is any one of the first dimension groups;

[0079] Specifically, any one first dimension group is selected, and the associated fields corresponding to the dimension fields in the first dimension group are determined, so as to obtain the above-mentioned multiple target associated fields.

[0080] Step S2042, determining step, determining a target metric field based on each of the target associated fields, wherein the target metric field is the metric field having the same field as the associated field;

[0081] Specifically, the target metric field is obtained by looking up the corresponding metric field in the fact table according to the target association field, and one dimension field corresponds to at least one target dimension field.

[0082] Step S2043, an integration step, adding the plurality of target metric fields to the target first dimension group to obtain the second dimension group;

[0083] Specifically, according to the corresponding relationship between the target metric field and the dimension field, the target metric field is added to the first dimension group to obtain the second dimension group.

[0084] Step S2044, repeating the above-mentioned obtaining step, the above-mentioned determining step, and the above-mentioned integrating step at least once in sequence until all the above-mentioned second dimension groups are obtained;

[0085] Specifically, the above-mentioned acquisition step, the above-mentioned determination step and the above-mentioned integration step complete the addition of the metric field of the above-mentioned first dimension group. The addition of the metric fields of all the above-mentioned first dimension groups can be completed by repeating the above-mentioned process.

[0086] Step S2045: setting corresponding operation functions according to the dimension fields and the metric fields in each of the second dimension groups to obtain multiple metric functions, wherein the metric functions include sum, minimum, maximum, average, count, standard deviation, and variance.

[0087] Specifically, the content described by the corresponding measurement field is determined based on the dimension field in the above-mentioned second dimension group. For example, a dimension field and its corresponding measurement field are used to describe the transaction volume of goods. The above-mentioned measurement function is obtained by determining the operation method for the numerical data included in the measurement field according to the analysis requirements.

[0088] Step S2046: construct a data model based on each of the second dimension groups and the corresponding metric function to obtain the corresponding cube.

[0089] Specifically, the metric function required for data analysis of each second dimension group is added to each second dimension group, and the modified second dimension group is constructed into a corresponding data model, that is, multiple cubes, namely the cubes, are obtained.

[0090] In order to complete the data analysis of each cube, in an optional implementation manner, the above step S205 includes:

[0091] Step S2051, a first construction step, constructing a grouping SQL statement according to the dimension fields in the target cube, where the target cube is any one of the above cubes;

[0092] Specifically, the description of the data analysis corresponding to the cube is converted into specific SQL statements to be executed in the data environment, including converting dimension fields into grouping fields of SQL statements, and each dimension field corresponds to a group of SQL statements.

[0093] Step S2052, the second construction step, constructs a value-taking SQL statement according to the above-mentioned metric function in the above-mentioned target cube;

[0094] Specifically, the description of the data analysis corresponding to the cube is converted into a specific SQL statement to be executed in the data environment, including converting the metric function into the value field of the SQL statement, setting the value range and operation method of the SQL statement according to the correspondence between the metric function and the dimension field, that is, configuring the corresponding SQL statement according to the operation method of the numerical value in the metric field corresponding to the dimension field.

[0095] Step S2053, a combining step, combining the above grouping SQL statement and the above value-taking SQL statement to obtain a corresponding target SQL statement;

[0096] Specifically, the corresponding connection statement is configured so that the above grouping SQL statement and the above value SQL statement are combined into a set of SQL statements that can complete the data analysis requirements of the above target cube, that is, the above target SQL statement is obtained.

[0097] Step S2054, repeating the first construction step, the second construction step, and the combination step at least once in sequence until the target SQL statements corresponding to all the cubes are obtained;

[0098] Specifically, the first construction step, the second construction step and the combination step complete the assembly of the SQL statement corresponding to one cube. By repeating the above steps, the assembly of the SQL statements corresponding to all cubes can be completed.

[0099] Step S2055 , executing each of the above target SQL statements in sequence to obtain the above data analysis results corresponding to each of the above cubes.

[0100] Specifically, the data analysis of each cube can be completed by sequentially running the above SQL statements in the CPU to obtain the data analysis results.

[0101] To facilitate querying the above data analysis results, in an optional implementation, after performing data analysis on each cube to obtain multiple data analysis results, the above method further includes:

[0102] Step S301: sort the above-mentioned dimension fields to obtain a sequence number corresponding to each of the above-mentioned dimension fields;

[0103] Specifically, the dimension fields in each of the above dimension tables are uniformly sorted and numbered, that is, a serial number covering all the above dimension fields is obtained.

[0104] Step S302: setting corresponding identification information for each of the serial numbers according to the dimension fields in each of the cubes, wherein the identification information is used to indicate whether the dimension field corresponding to the serial number exists in the cube;

[0105] Specifically, depending on whether the cube corresponding to each data analysis result contains the dimension field corresponding to the serial number, the identification information corresponding to the serial number is set to 1 or 0, where 1 indicates existence and 0 indicates non-existence, and the dimension field contained in the cube is represented in the form of serial number and identification information, saving storage resources.

[0106] Step S303: Generate a first row of key values ​​based on the identification information, and generate a second row of key values ​​based on the data analysis results;

[0107] Specifically, the first part of the row key value is generated according to the serial number and the identification information, and then the calculation result of the metric field corresponding to the corresponding dimension field is written into the corresponding position corresponding to the serial number, so as to obtain the second part of the row key value.

[0108] Step S304: Combine the first row key value and the second row key value to obtain a target row key value, and store the target row key value in a corresponding position in the distributed cluster.

[0109] Specifically, a row key representing the data analysis results is formed from the first and second row keys. A sequence number is added to the row key to obtain the target row key, which is then stored in the corresponding location in the distributed cluster based on the sequence number. Storing the data analysis results in the form of row keys improves the efficiency of sequential queries and enhances query concurrency through the scalability of distributed columnar storage.

[0110] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the multidimensional data analysis method of the present application will be described in detail below with reference to specific embodiments.

[0111] This embodiment relates to a specific method for analyzing multidimensional data. Figure 4 As shown, the following steps are included:

[0112] Step S1: According to business analysis requirements, determine all data tables that need to be accessed for analysis;

[0113] Step S2: Based on the data table, determine the fact table and dimension table as well as the associated fields, and build a star model;

[0114] Step S3: Divide the fields in the fact table and dimension table into dimension fields, measure fields, and meaningless fields;

[0115] Step S4: Analyze mutually exclusive fields and similar fields in the dimension fields according to business query requirements;

[0116] Step S5: Divide the dimension fields into multiple dimension groups based on mutually exclusive fields and similar fields;

[0117] Step S6: Analyze the characteristics of the fact records described by the dimension fields in each dimension group and divide them into different hierarchical dimensions;

[0118] Step S7: Add corresponding metric fields and metric functions for each dimension group;

[0119] Step S8: Construct a corresponding SQL statement based on the combination of dimension fields, metric fields, and metric functions in each dimension group;

[0120] Step S9: Set the data analysis cycle to ensure that the corresponding data analysis is completed before each data query, and store the analysis results in a distributed manner;

[0121] Step S10: According to the data query requirements, the corresponding data analysis results are retrieved and obtained.

[0122] In addition, when there is a query request other than the data analysis result in the data query requirement, data analysis is performed on the data corresponding to the query request according to the original data table to ensure that the above data query requirement is completed.

[0123] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0124] The present application also provides a data analysis device for multidimensional data. It should be noted that the data analysis device for multidimensional data of the present application embodiment can be used to execute the data analysis method for multidimensional data provided by the present application embodiment. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the details that have been described will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0125] The following introduces the data analysis device for multidimensional data provided in the embodiments of the present application.

[0126] Figure 5 : is a structural block diagram of a multidimensional data analysis device according to an embodiment of the present application. Figure 5 As shown, the device includes:

[0127] An acquisition unit 10 is configured to acquire a target data table and analyze the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table.

[0128] Specifically, according to the analysis requirements of data analysis, all data tables that need to be accessed in the database, namely the above-mentioned target data tables, are determined, and then various data recorded in the data tables are analyzed, extracted and classified, namely the above-mentioned fact tables and the above-mentioned dimension tables are obtained.

[0129] In one embodiment, there is only one fact table in the same star schema, and a dimension table may or may not exist. In this embodiment, multiple dimension tables exist at the same time.

[0130] A first analysis unit 20 is configured to analyze the fact table and each dimension table to obtain a plurality of dimension fields and a plurality of metric fields, wherein the dimension fields are fields for describing characteristics of the event records, and the metric fields are fields for numerical values ​​in the event records, and each dimension field corresponds to one metric field;

[0131] Specifically, the fields in the fact table and the dimension table are divided into the dimension fields and the metric fields according to the contents contained in the fields.

[0132] In a specific implementation, the dimension fields mentioned above may come from a fact table or a dimension table, and the metric fields mentioned above only come from a fact table.

[0133] A second analysis unit 30 is configured to analyze each of the dimension fields to determine a first association type between the dimension fields and to combine the dimension fields according to the first association type to obtain a plurality of first dimension groups, wherein the first association type includes similar fields, mutually exclusive fields, and unrelated fields, and the first association type between any two dimension fields in the first dimension group is neither similar fields nor mutually exclusive fields;

[0134] Specifically, semantic analysis is performed on the above-mentioned dimension fields and the above-mentioned measurement fields to determine similar fields, and based on the data analysis requirements and field semantics, the association types between fields that will not be analyzed together during the data analysis process are determined as mutually exclusive fields, and the association types between the remaining fields are determined as unrelated fields. In the process of dimension combination of fields, there are no two dimension fields in the same dimension combination that are similar or mutually exclusive fields. Figure 3 As shown, the dimension fields that are meaningless to each other are combined to obtain a plurality of first dimension groups, and there may be overlap or no intersection between the groups.

[0135] A construction unit 40 is configured to add the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain a plurality of second dimension groups, and to construct corresponding data models according to each of the second dimension groups to obtain a plurality of cubes;

[0136] Specifically, the metric fields corresponding to the dimension fields in the above-mentioned first dimension combinations are determined, and then they are associated one by one to obtain the above-mentioned second dimension group, and then the corresponding data model is constructed according to the dimension fields and metric fields in the second dimension group to obtain the above-mentioned cube.

[0137] The third analysis unit 50 is used to perform data analysis on each cube to obtain multiple data analysis results.

[0138] Specifically, data operations are performed on the metric fields corresponding to each dimension field, including summation, maximum value, minimum value statistics, etc. The above operation results are the data analysis results.

[0139] Through this embodiment, the acquisition unit acquires the target data table, and analyzes the target data table to obtain a fact table and multiple dimension tables, the target data table is the data table to be analyzed, the fact table is used to store event records, and the dimension table is used to store the characteristics of the event records in the fact table; the first analysis unit obtains multiple dimension fields and multiple measurement fields based on the analysis of the fact table and the dimension tables, the dimension fields are fields for describing the characteristics of the event records, and the measurement fields are fields for numerical values ​​in the event records, and one dimension field corresponds to one measurement field; the second analysis unit analyzes each dimension field to determine the characteristics of each dimension field. The first association type between dimension fields and the combination of the dimension fields according to the first association type to obtain multiple first dimension groups, the first association type includes similar fields, mutually exclusive fields and unrelated fields, and the first association type between any two dimension fields in the first dimension group is not the similar field or the mutually exclusive field; the construction unit adds the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension group to obtain multiple second dimension groups, and constructs corresponding data models according to each of the second dimension groups to obtain multiple cubes; the third analysis unit performs data analysis on each cube to obtain multiple data analysis results. The present application performs a meaning analysis on the dimension fields in the data model, determines that the fields with similar functions in the data analysis are similar fields, and the fields that should not be analyzed at the same time are mutually exclusive fields. When combining dimensions, the dimensions and corresponding metrics of the similar fields and mutually exclusive fields are set not to exist in the same combination, which greatly reduces unnecessary dimension combinations and solves the problem in the prior art that when performing data analysis on the data model, all dimensions are combined, resulting in excessive calculation and excessive resource occupation.

[0140] In order to sort out the data fields in the target data table, in an optional embodiment, the first analysis unit includes:

[0141] a first comparison module, configured to compare fields in the fact table and each dimension table to obtain a plurality of associated fields and meaningless fields, wherein the associated fields are the same fields in the dimension table and the fact table, and the meaningless fields are the fields in the fact table and the dimension table used to represent system status control and configuration control;

[0142] Specifically, the fields stored in the fact table and the dimension table are compared, and the same fields therein are traversed to determine as associated fields. The fact table and the dimension table are linked through the associated fields, and there is no association between the dimension tables.

[0143] In practice, fact tables and dimension tables should theoretically contain only fields that store fact records and describe their characteristics, namely, dimension fields and metric fields. However, in practice, fact tables and dimension tables often contain many fields that are irrelevant to the business, such as those for system status control and configuration control data change events. These fields are not meaningful for business analysis, so they are classified as meaningless. When building the data model, these meaningless fields are eliminated to reduce the possibility of dimension combinations and computational complexity.

[0144] a first determining module, configured to determine a second association type between the dimension table and the fact table based on a proportion of the association fields in each dimension table, wherein the second association type includes an inner join, a left outer join, a right outer join, and a full outer join, and each second association type corresponds to each dimension table in a one-to-one manner;

[0145] Specifically, the association type between the fact table and the dimension table is determined based on the proportion of the fields to which the associated fields belong in all the fields included in the table. For example, based on the proportion in the fact table, when the proportion in the dimension table is higher than that in the fact table, the left outer join association type is selected, that is, the fields to which the above-mentioned associated fields belong in the fact table are retained, and the remaining fields are eliminated to ensure that when a cube is built based on the dimension table, all metric fields have corresponding dimension fields.

[0146] a deletion module, configured to delete the meaningless fields in the fact table and each dimension table, and construct a star data model based on the fact table, each dimension table, and each second association type;

[0147] Specifically, the above meaningless fields are deleted, and fields in the fact table or dimension table are removed according to the above second association type between each dimension table and the fact table, and the modified fact table and dimension table are associated according to the associated fields to obtain the above star data model.

[0148] The first analysis module is used to analyze each of the above fields in the above star data model to obtain the multiple above dimension fields and the above metric fields.

[0149] Specifically, the fields in the star data model are analyzed and divided into dimension fields and measure fields according to the content described by the fields.

[0150] In order to simplify the amount of calculation in the data analysis process, in an optional embodiment, the second analysis unit includes:

[0151] A second determining module is configured to perform semantic analysis on each of the dimension fields, and determine the first association type between the dimension fields whose corresponding semantic similarity exceeds a first threshold as the similar field;

[0152] Specifically, a semantic analysis is performed on each dimension field from the perspective of business meaning, and two dimension fields with similar semantics are determined to be similar fields, such as region and regional scale. These two fields have similar business meanings. Putting them in the same dimension combination for data analysis will only slightly improve the effect and significantly increase the number of combinations and computational complexity. Therefore, similar fields in this application are not combined when combining dimensions.

[0153] a third determining module, configured to consult a preset information table to obtain the dimension fields corresponding to the mutually exclusive fields of the first association type, the preset information table including the dimension fields that are not combined in the data analysis requirement;

[0154] Specifically, there are preset information tables corresponding to different data analysis requirements, which include field combinations that will not improve the data analysis effect after being combined during the data analysis process. By referring to the above information tables, it is possible to determine which dimension fields in the current star data model have association types that are mutually exclusive fields.

[0155] The fourth determining module is configured to determine the first association type between the dimension fields whose first association type is not the similar field and the mutually exclusive field as an unrelated field.

[0156] Specifically, after determining that the field corresponding to the above first association type is a similar field or a mutually exclusive field, the above first association type between it and the field is determined to be an unrelated field, that is, a field that can be freely combined in dimensions during data analysis.

[0157] In the above embodiment, there are two fields that are mutually exclusive or similar. These two dimension fields can be combined with the remaining n dimension fields for multidimensional analysis. n+1 possible combinations, but in traditional multidimensional data analysis methods, the first type of association is not divided. n+2 possible combinations, the effect of the method of this application is no different from the existing technology, but when there are three fields that are mutually exclusive or similar, these three dimension fields can be combined with the remaining n-1 dimension fields for multidimensional analysis. n possible combinations, which is 2 fewer than traditional multidimensional data analysis methods. n When considering each possible combination, it can be inferred that the more fields corresponding to the first association type are mutually exclusive fields or similar fields, the more obvious the effect of the method of the present application on reducing the amount of calculation is.

[0158] In order to simplify the amount of calculation in the data analysis process, in an optional embodiment, the second analysis unit further includes:

[0159] A classification module is used to analyze each of the above-mentioned dimension fields and classify the above-mentioned dimension fields corresponding to the same above-mentioned characteristics into the same level to obtain multiple dimension levels;

[0160] Specifically, the aforementioned dimensional hierarchy is obtained by analyzing the characteristics of the event records described by the dimension fields and grouping the dimensions used to record the same characteristics into the same level. Each level includes at least one dimension field. For example, the time dimensions year, quarter / month, and day; the regional dimensions province, city, and county; and the organizational dimensions first-level, second-level, and third-level institutions can all be categorized into the same dimensional level.

[0161] The first combining module is configured to combine the dimension fields according to the first association type and the dimension level to obtain a plurality of the first dimension groups, wherein any two dimension fields in the first dimension groups are not in the same dimension level.

[0162] In the embodiment of the present application, the purpose of setting the dimension hierarchy is to reduce the possibility of dimension combinations, that is, to combine the dimension combinations as a whole, that is, to select only one dimension in the dimension hierarchy for combination.

[0163] In order to establish a data model from a dimensional perspective to facilitate data analysis, in an optional embodiment, the construction unit includes:

[0164] an acquisition module, configured to execute the acquisition step to acquire a plurality of target associated fields, wherein the target associated fields are associated fields that have the same fields as the dimension fields in the target first dimension group, and the target first dimension group is any one of the first dimension groups;

[0165] Specifically, any one first dimension group is selected, and the associated fields corresponding to the dimension fields in the first dimension group are determined, so as to obtain the above-mentioned multiple target associated fields.

[0166] a fifth determining module, configured to execute the determining step, and determine a target metric field according to each of the target associated fields, wherein the target metric field is the metric field having the same field as the associated field;

[0167] Specifically, the target metric field is obtained by looking up the corresponding metric field in the fact table according to the target association field, and one dimension field corresponds to at least one target dimension field.

[0168] A first integration module is configured to perform an integration step, adding the plurality of target metric fields to the target first dimension group to obtain the second dimension group;

[0169] Specifically, according to the corresponding relationship between the target metric field and the dimension field, the target metric field is added to the first dimension group to obtain the second dimension group.

[0170] a first repeating module, configured to sequentially repeat the obtaining step, the determining step, and the integrating step at least once until all the second dimension groups are obtained;

[0171] Specifically, the above-mentioned acquisition step, the above-mentioned determination step and the above-mentioned integration step complete the addition of the metric field of the above-mentioned first dimension group. The addition of the metric fields of all the above-mentioned first dimension groups can be completed by repeating the above-mentioned process.

[0172] A first integration module is configured to set corresponding operation functions according to the dimension fields and the metric fields in each of the second dimension groups to obtain a plurality of metric functions, wherein the metric functions include sum, minimum, maximum, average, count, standard deviation, and variance.

[0173] Specifically, the content described by the corresponding measurement field is determined based on the dimension field in the above-mentioned second dimension group. For example, a dimension field and its corresponding measurement field are used to describe the transaction volume of goods. The above-mentioned measurement function is obtained by determining the operation method for the numerical data included in the measurement field according to the analysis requirements.

[0174] The first construction module is used to construct a data model according to each of the second dimension groups and the corresponding metric function to obtain the corresponding cube.

[0175] Specifically, the metric function required for data analysis of each second dimension group is added to each second dimension group, and the modified second dimension group is constructed into a corresponding data model, that is, multiple cubes, namely the cubes, are obtained.

[0176] In order to complete the data analysis of each cube, in an optional implementation manner, the third analysis unit includes:

[0177] A second building module is configured to execute the first building step and build a grouping SQL statement according to the dimension fields in the target cube, where the target cube is any one of the above cubes;

[0178] Specifically, the description of the data analysis corresponding to the cube is converted into specific SQL statements to be executed in the data environment, including converting dimension fields into grouping fields of SQL statements, and each dimension field corresponds to a group of SQL statements.

[0179] A third construction module is used to execute the second construction step and construct a value-taking SQL statement according to the metric function in the target cube;

[0180] Specifically, the description of the data analysis corresponding to the cube is converted into a specific SQL statement to be executed in the data environment, including converting the metric function into the value field of the SQL statement, setting the value range and operation method of the SQL statement according to the correspondence between the metric function and the dimension field, that is, configuring the corresponding SQL statement according to the operation method of the numerical value in the metric field corresponding to the dimension field.

[0181] A second combining module is used to execute the combining step to combine the above-mentioned grouping SQL statement and the above-mentioned value SQL statement to obtain a corresponding target SQL statement;

[0182] Specifically, the corresponding connection statement is configured so that the above grouping SQL statement and the above value SQL statement are combined into a set of SQL statements that can complete the data analysis requirements of the above target cube, that is, the above target SQL statement is obtained.

[0183] A second repeating module is used to sequentially repeat the first building step, the second building step, and the combining step at least once until the target SQL statements corresponding to all the cubes are obtained;

[0184] Specifically, the first construction step, the second construction step and the combination step complete the assembly of the SQL statement corresponding to one cube. By repeating the above steps, the assembly of the SQL statements corresponding to all cubes can be completed.

[0185] The execution module is used to execute the above target SQL statements in sequence to obtain the above data analysis results corresponding to the above cubes.

[0186] Specifically, the data analysis of each cube can be completed by sequentially running the above SQL statements in the CPU to obtain the data analysis results.

[0187] In order to facilitate querying the above data analysis results, in an optional embodiment, the above device further includes:

[0188] A sorting unit, configured to sort the dimension fields to obtain a sequence number corresponding to the dimension fields;

[0189] Specifically, the dimension fields in each of the above dimension tables are uniformly sorted and numbered, that is, a serial number covering all the above dimension fields is obtained.

[0190] a setting unit, configured to set corresponding identification information for each of the serial numbers according to the dimension fields in each of the cubes, wherein the identification information is used to indicate whether the dimension field corresponding to the serial number exists in the cube;

[0191] Specifically, depending on whether the cube corresponding to each data analysis result contains the dimension field corresponding to the serial number, the identification information corresponding to the serial number is set to 1 or 0, where 1 indicates existence and 0 indicates non-existence, and the dimension field contained in the cube is represented in the form of serial number and identification information, saving storage resources.

[0192] A generating unit, configured to generate a first row of key values ​​according to the identification information, and generate a second row of key values ​​according to the data analysis result;

[0193] Specifically, the first part of the row key value is generated according to the serial number and the identification information, and then the calculation result of the metric field corresponding to the corresponding dimension field is written into the corresponding position corresponding to the serial number, so as to obtain the second part of the row key value.

[0194] The combining unit is configured to combine the first row key value and the second row key value to obtain a target row key value, and store the target row key value in a corresponding position in the distributed cluster.

[0195] Specifically, a row key representing the data analysis results is formed from the first and second row keys. A sequence number is added to the row key to obtain the target row key, which is then stored in the corresponding location in the distributed cluster based on the sequence number. Storing the data analysis results in the form of row keys improves the efficiency of sequential queries and enhances query concurrency through the scalability of distributed columnar storage.

[0196] The multidimensional data analysis device includes a processor and a memory. The acquisition unit, first analysis unit, second analysis unit, construction unit, and third analysis unit are all stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. The modules are all located in the same processor; alternatively, the modules can be located in different processors in any combination.

[0197] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to reduce the computational complexity of the data analysis process.

[0198] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0199] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program is executed, the device where the computer-readable storage medium is located is controlled to execute the multi-dimensional data analysis method.

[0200] Specifically, the data analysis methods for multidimensional data include:

[0201] Step S201: Obtain a target data table and analyze the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table.

[0202] Step S202: Analyze the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields. The dimension fields are fields used to describe characteristics of the event records, and the metric fields are fields containing numerical values ​​in the event records. One dimension field corresponds to one metric field.

[0203] Step S203: Analyze each of the dimension fields to determine a first association type between each of the dimension fields, and combine the dimension fields according to the first association type to obtain multiple first dimension groups. The first association type includes similar fields, mutually exclusive fields, and unrelated fields. The first association type between any two of the dimension fields in the first dimension group is neither similar fields nor mutually exclusive fields.

[0204] Step S204: adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain multiple second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain multiple cubes;

[0205] Step S205 , performing data analysis on each cube to obtain multiple data analysis results.

[0206] An embodiment of the present invention provides a processor, which is used to run a program, wherein the multi-dimensional data analysis method is executed when the program is run.

[0207] Specifically, the data analysis methods for multidimensional data include:

[0208] Step S201: Obtain a target data table and analyze the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table.

[0209] Step S202: Analyze the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields. The dimension fields are fields used to describe characteristics of the event records, and the metric fields are fields containing numerical values ​​in the event records. One dimension field corresponds to one metric field.

[0210] Step S203: Analyze each of the dimension fields to determine a first association type between each of the dimension fields, and combine the dimension fields according to the first association type to obtain multiple first dimension groups. The first association type includes similar fields, mutually exclusive fields, and unrelated fields. The first association type between any two of the dimension fields in the first dimension group is neither similar fields nor mutually exclusive fields.

[0211] Step S204: adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain multiple second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain multiple cubes;

[0212] Step S205 , performing data analysis on each cube to obtain multiple data analysis results.

[0213] An embodiment of the present invention provides a data analysis system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0214] Step S201: Obtain a target data table and analyze the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table.

[0215] Step S202: Analyze the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields. The dimension fields are fields used to describe characteristics of the event records, and the metric fields are fields containing numerical values ​​in the event records. One dimension field corresponds to one metric field.

[0216] Step S203: Analyze each of the dimension fields to determine a first association type between each of the dimension fields, and combine the dimension fields according to the first association type to obtain multiple first dimension groups. The first association type includes similar fields, mutually exclusive fields, and unrelated fields. The first association type between any two of the dimension fields in the first dimension group is neither similar fields nor mutually exclusive fields.

[0217] Step S204: adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain multiple second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain multiple cubes;

[0218] Step S205 , performing data analysis on each cube to obtain multiple data analysis results.

[0219] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0220] Step S201: Obtain a target data table and analyze the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table.

[0221] Step S202: Analyze the fact table and each dimension table to obtain multiple dimension fields and multiple metric fields. The dimension fields are fields used to describe characteristics of the event records, and the metric fields are fields containing numerical values ​​in the event records. One dimension field corresponds to one metric field.

[0222] Step S203: Analyze each of the dimension fields to determine a first association type between each of the dimension fields, and combine the dimension fields according to the first association type to obtain multiple first dimension groups. The first association type includes similar fields, mutually exclusive fields, and unrelated fields. The first association type between any two of the dimension fields in the first dimension group is neither similar fields nor mutually exclusive fields.

[0223] Step S204: adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain multiple second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain multiple cubes;

[0224] Step S205 , performing data analysis on each cube to obtain multiple data analysis results.

[0225] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, can be centralized on a single computing device, or can be distributed across a network of multiple computing devices. They can be implemented using program code executable by the computing device, and thus, can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described herein can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0226] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0227] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0228] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0229] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0230] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0231] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0232] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0233] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0234] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0235] 1) The data analysis method for multidimensional data of the present application first obtains a target data table and analyzes the target data table to obtain a fact table and multiple dimension tables. The target data table is the data table to be analyzed. The fact table is used to store event records, and the dimension table is used to store the characteristics of the event records in the fact table. Then, based on the fact table and each dimension table, multiple dimension fields and multiple measurement fields are obtained. The dimension fields are fields used to describe the characteristics of the event records, and the measurement fields are fields of numerical values ​​in the event records. One dimension field corresponds to one measurement field. Then, each dimension field is analyzed. Determine the first association type between each of the above-mentioned dimension fields and combine the above-mentioned dimension fields according to the above-mentioned first association type to obtain multiple first dimension groups, the above-mentioned first association type includes similar fields, mutually exclusive fields and unrelated fields, and the above-mentioned first association type between any two of the above-mentioned dimension fields in the above-mentioned first dimension group is not the above-mentioned similar field or the above-mentioned mutually exclusive field; then, add the above-mentioned measurement fields corresponding to the above-mentioned dimension fields in each of the above-mentioned first dimension groups to the corresponding above-mentioned first dimension group to obtain multiple second dimension groups, and construct corresponding data models according to each of the above-mentioned second dimension groups to obtain multiple cubes; finally, perform data analysis on each cube to obtain multiple data analysis results. The present application performs a meaning analysis on the dimension fields in the data model, determines that the fields with similar functions in the data analysis are similar fields, and the fields that should not be analyzed at the same time are mutually exclusive fields, and sets the dimensions and corresponding measurements of the mutually similar fields and mutually exclusive fields to not exist in the same combination when combining dimensions, which greatly reduces unnecessary dimension combinations and solves the problem in the prior art that when performing data analysis on the data model, all dimensions are combined, resulting in excessive calculation and excessive resource occupation.

[0236] 2) The data analysis device for multidimensional data of the present application, the acquisition unit acquires a target data table, and analyzes the target data table to obtain a fact table and multiple dimension tables, the target data table is a data table to be analyzed, the fact table is used to store event records, and the dimension table is used to store the characteristics of the event records in the fact table; the first analysis unit analyzes the fact table and each dimension table to obtain multiple dimension fields and multiple measurement fields, the dimension field is a field for describing the characteristics of the event record, the measurement field is a field for the numerical value in the event record, and one dimension field corresponds to one measurement field; the second analysis unit analyzes each dimension field The analysis determines the first association type between each of the above-mentioned dimension fields and combines the above-mentioned dimension fields according to the above-mentioned first association type to obtain multiple first dimension groups. The above-mentioned first association type includes similar fields, mutually exclusive fields and unrelated fields. The above-mentioned first association type between any two of the above-mentioned dimension fields in the above-mentioned first dimension group is not the above-mentioned similar field or the above-mentioned mutually exclusive field; the construction unit adds the above-mentioned measurement fields corresponding to the above-mentioned dimension fields in each of the above-mentioned first dimension groups to the corresponding above-mentioned first dimension group to obtain multiple second dimension groups, and constructs corresponding data models according to each of the above-mentioned second dimension groups to obtain multiple cubes; the third analysis unit performs data analysis on each cube to obtain multiple data analysis results. The present application performs a meaning analysis on the dimension fields in the data model, determines that the fields with similar functions in the data analysis are similar fields, and the fields that should not be analyzed at the same time are mutually exclusive fields. When combining dimensions, the dimensions and corresponding measurements of the mutually similar fields and mutually exclusive fields are set not to exist in the same combination, which greatly reduces unnecessary dimension combinations and solves the problem in the prior art that when performing data analysis on the data model, the dimension combination of all dimensions leads to excessive calculation and excessive resource occupation.

[0237] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A data analysis method for multidimensional data, characterized in that: include: Obtaining a target data table and analyzing the target data table to obtain a fact table and multiple dimension tables, wherein the target data table is the data table to be analyzed, the fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table; Analyzing the fact table and each dimension table to obtain a plurality of dimension fields and a plurality of metric fields, wherein the dimension fields are fields for describing characteristics of the event records, and the metric fields are fields of numerical values ​​in the event records, and one dimension field corresponds to one metric field; Analyzing each of the dimension fields to determine a first association type between each of the dimension fields, and combining the dimension fields according to the first association type to obtain a plurality of first dimension groups, wherein the first association type includes similar fields, mutually exclusive fields, and unrelated fields, and the first association type between any two of the dimension fields in the first dimension group is not the similar field or the mutually exclusive field; Adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension group to obtain a plurality of second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain a plurality of cubes; Data analysis is performed on each cube to obtain multiple data analysis results.

2. The method according to claim 1, characterized in that Multiple dimension fields and multiple metric fields are obtained based on the analysis of the fact table and each dimension table, including: Comparing the fields in the fact table and each dimension table to obtain a plurality of associated fields and meaningless fields, wherein the associated fields are the same fields in the dimension table as in the fact table, and the meaningless fields are the fields in the fact table and the dimension table used to represent system status control and configuration control; determining a second association type between the dimension table and the fact table based on a proportion of the association fields in each dimension table, wherein the second association type includes an inner join, a left outer join, a right outer join, and a full outer join, and the second association type corresponds to the dimension table in a one-to-one manner; Deleting the meaningless fields in the fact table and each dimension table, and constructing a star data model based on the fact table, each dimension table and each second association type; Each of the fields in the star data model is analyzed to obtain a plurality of dimension fields and the metric fields.

3. The method according to claim 1, characterized in that Analyzing each of the dimension fields to determine a first association type between the dimension fields includes: Performing semantic analysis on each of the dimension fields, and determining the first association type between the dimension fields whose corresponding semantic similarity exceeds a first threshold as the similar field; Obtaining the dimension field corresponding to the mutually exclusive field of the first association type by consulting a preset information table, wherein the preset information table includes the dimension fields that are not combined in the data analysis requirement; The first association type between the dimension fields whose first association type is not the similarity field and the mutually exclusive field is determined as an unassociated field.

4. The method according to claim 1, wherein Combining the dimension fields according to the first association type to obtain multiple first dimension groups includes: Analyze each of the dimension fields and classify the dimension fields corresponding to the same characteristic into the same level to obtain multiple dimension levels; The dimension fields are combined according to the first association type and the dimension level to obtain a plurality of first dimension groups, and any two dimension fields in the first dimension groups are not in the same dimension level.

5. The method according to claim 2, characterized in that Adding the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension groups to obtain multiple second dimension groups, and constructing corresponding data models according to each of the second dimension groups to obtain multiple cubes, including: an acquisition step of acquiring a plurality of target associated fields, wherein the target associated fields are associated fields that have the same fields as the dimension fields in the target first dimension group, and the target first dimension group is any one of the first dimension groups; a determining step of determining a target metric field according to each of the target associated fields, wherein the target metric field is the metric field having the same field as the associated field; an integration step, adding a plurality of the target metric fields to the target first dimension group to obtain the second dimension group; Repeating the obtaining step, the determining step, and the integrating step at least once in sequence until all the second dimension groups are obtained; Setting corresponding operation functions according to the dimension fields and the metric fields in each of the second dimension groups to obtain multiple metric functions, wherein the metric functions include sum, minimum, maximum, average, count, standard deviation, and variance; A data model is constructed according to each of the second dimension groups and the corresponding metric function to obtain the corresponding cube.

6. The method according to claim 5, characterized in that Data analysis of each cube yields multiple data analysis results, including: A first construction step is to construct a grouping SQL statement according to the dimension field in the target cube, where the target cube is any one of the cubes; The second construction step is to construct a value-taking SQL statement according to the metric function in the target cube; A combining step, combining the grouping SQL statement and the value-taking SQL statement to obtain a corresponding target SQL statement; Repeat the first construction step, the second construction step, and the combination step at least once in sequence until the target SQL statements corresponding to all the cubes are obtained; Execute the target SQL statements in sequence to obtain the data analysis results corresponding to the cubes.

7. The method according to claim 1, characterized in that After performing data analysis on each cube to obtain multiple data analysis results, the method further includes: Sorting each dimension field to obtain a sequence number corresponding to each dimension field; Setting corresponding identification information for each of the serial numbers according to the dimension fields in each of the cubes, the identification information being used to indicate whether the dimension field corresponding to the serial number exists in the cube; Generate a first row of key values ​​according to the identification information, and generate a second row of key values ​​according to the data analysis result; The first row key value and the second row key value are combined to obtain a target row key value, and the target row key value is stored in a corresponding position in the distributed cluster.

8. A data analysis device for multidimensional data, characterized in that: The device comprises: an acquisition unit, configured to acquire a target data table and analyze the target data table to obtain a fact table and multiple dimension tables, wherein the target data table is a data table to be analyzed, the fact table is used to store event records, and the dimension tables are used to store characteristics of the event records in the fact table; a first analyzing unit, configured to analyze the fact table and each dimension table to obtain a plurality of dimension fields and a plurality of metric fields, wherein the dimension fields are fields for describing characteristics of the event records, and the metric fields are fields for numerical values ​​in the event records, and one dimension field corresponds to one metric field; a second analyzing unit, configured to analyze each dimension field to determine a first association type between each dimension field, and to combine the dimension fields according to the first association type to obtain a plurality of first dimension groups, wherein the first association type includes similar fields, mutually exclusive fields, and unrelated fields, and the first association type between any two dimension fields in the first dimension group is neither the similar fields nor the mutually exclusive fields; a construction unit, configured to add the metric fields corresponding to the dimension fields in each of the first dimension groups to the corresponding first dimension group to obtain a plurality of second dimension groups, and to construct a corresponding data model according to each of the second dimension groups to obtain a plurality of cubes; The third analysis unit is used to perform data analysis on each cube to obtain multiple data analysis results.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A data analysis system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method of any one of claims 1 to 7.

Images