Data insight apparatus oriented to exploratory data analysis

Abstract

The application provides a data insight device for exploratory data analysis, having the features of comprising an offline construction module for constructing a corresponding mode cube according to a target data set; an online analysis module for calculating an insight result according to an analysis instruction and the mode cube, wherein the online analysis module comprises: a data range generation unit for generating data ranges; a homogeneous data range generation unit for generating homogeneous data ranges corresponding to the data ranges; an online mode calculation unit for calculating data modes corresponding to the homogeneous data ranges according to the mode cube; an insight generation unit for performing co-analyzing the data modes of the homogeneous data ranges to obtain corresponding insights; and a recommendation unit for performing value analysis on all the insights to generate the insight result. In summary, the device can avoid repeated calculation of data range modes and improve insight discovery efficiency.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, specifically to a data insight device for exploratory data analysis. Background Technology

[0002] With the continuous development of communication and information technologies, the rate at which data is generated is increasing rapidly. Simultaneously, with the evolution and development of database and storage technologies, more and more data is being recorded and stored for future development and utilization. Internet search engines support billions of web searches, processing tens of thousands of terabytes of data daily. The ever-increasing volume of data, the deepening applications, and the undeniable value of data compel us to explore how we can better benefit from it.

[0003] To help analysts discover the inherent patterns in data, Exploratory Data Analysis (EDA) was proposed. When data analysts face a completely new and unknown dataset, understanding the information and characteristics contained within the dataset during the exploration process to gain basic "insights" becomes particularly important. Currently, the industry has many excellent exploratory data analysis tools or platforms to assist users, such as Pandas, Matplotlib, Microsoft Power BI, and Tableau. As the volume of data gradually increases, from a system perspective, the analysis system needs longer execution time to explore insights from the data; from a user perspective, large datasets can easily cause users to "lose focus," as users may only be interested in a portion of the data within a large dataset.

[0004] Therefore, how to quickly explore data that users are interested in, and how to provide analysts with a fast and accurate way to gain valuable insights are important issues that urgently need to be addressed. Summary of the Invention

[0005] This invention was made to solve the above-mentioned problems, and its purpose is to provide a data insight device for exploratory data analysis.

[0006] This invention provides a data insight device for exploratory data analysis, used to obtain insight results for a target dataset according to user analysis instructions. It features: an offline construction module for constructing a corresponding pattern cube based on the target dataset; and an online analysis module for calculating the insight results based on the analysis instructions and the pattern cube. The data in the target dataset is divided into measurement attributes and dimensional attributes. Measurement attribute data is numerical, while dimensional attribute data is non-numerical. The offline construction module includes: a grouping unit for grouping all dimensional attributes and combining each grouping result with each measurement attribute to obtain multiple dimensional attribute combinations; an aggregation unit for aggregating the dimensional attribute combinations using various aggregation methods based on the corresponding data in the target dataset to obtain corresponding data cells; and a single-value data range generation unit for generating single-value data ranges for each dimension. The system combines degree attribute combinations to perform single-value splitting and recombination to obtain corresponding single-value data ranges. An offline pattern calculation unit performs data pattern calculations on each single-value data range to obtain corresponding pattern cells. An offline storage unit stores all data cells and pattern cells as a pattern cube. The online analysis module includes: a data range generation unit, which generates at least one corresponding data range based on analysis instructions and the pattern cube; a homogeneous data range generation unit, which generates homogeneous data ranges corresponding to each data range; an online pattern calculation unit, which calculates the data pattern corresponding to each homogeneous data range based on the pattern cube; an insight generation unit, which performs commonality analysis on the data patterns of each homogeneous data range to obtain corresponding insights; and a recommendation unit, which performs value analysis on all insights and generates the top-k most valuable insights as the insight results.

[0007] The data insight device for exploratory data analysis provided by the present invention may also have the following feature: the aggregation method includes "SUM", "MAX", "AVG" and "MIN".

[0008] The data insight device for exploratory data analysis provided by this invention may also have the following feature: wherein the expression of the data cell is: cell data = < a1, a2, ... a i :value>, where cell data Let i be the data cell. data The total number of dimension attributes in the corresponding dimension attribute combination, a i This refers to the specific value of the i-th dimension attribute in this dimension attribute combination, where value is the data cell. data The aggregation result of the measurement attributes in the corresponding dimensional attribute combination after applying the corresponding aggregation method.

[0009] The data insight device for exploratory data analysis provided by this invention may also have the following feature: wherein the expression of the pattern cell is: cell pattern = < a1, a2, ... a i : pattern_dict>, where cell pattern Let i be the pattern cell. pattern The total number of dimension attributes in the corresponding single-value data range, a i For the i-th dimension attribute within this single-value data range, pattern_dict represents the pattern cell. pattern The data pattern result is obtained by performing data pattern calculation on the corresponding single-value data range. agg(m) is the aggregation result after agg is applied to the measurement attribute m. The pattern cell has one and only one dimension attribute with the value "ALL". "ALL" represents all possible values ​​of that dimension attribute.

[0010] The data insight device for exploratory data analysis provided by the present invention may also have the following features: wherein the analysis instruction includes the user's value for at least one dimension attribute, the data range is a single-value data range and a multi-value data range including the value, and the multi-value data range is obtained by selecting the corresponding single-value data range from the pattern cube and performing an aggregation operation.

[0011] The data insight device for exploratory data analysis provided by the present invention may also have the following features: wherein the homogeneous data range includes multiple sub-data ranges, the sub-data ranges being the data ranges corresponding to the homogeneous data ranges and the data ranges obtained by expanding the values ​​of the dimensional attributes of the data ranges.

[0012] The data insight device for exploratory data analysis provided by this invention may also have the following features: the online pattern calculation unit includes: a first calculation subunit, used to calculate the data pattern in the corresponding pattern cell as the data pattern of a sub-data range belonging to a single-value data range; and a second calculation subunit, storing pruning inferences, used to calculate the data pattern of a sub-data range belonging to a multi-value data range based on the relevant single-value data range and pruning inferences.

[0013] The data insight device for exploratory data analysis provided by this invention may also have the following features: the second calculation subunit generates data patterns by the following steps: step S1, obtaining inference pattern results based on the data pattern calculation results and pruning inferences corresponding to the relevant single-value data range; step S2, removing the patterns to be analyzed corresponding to the inference pattern results from all patterns to be analyzed to obtain the patterns to be calculated; step S3, calculating the patterns to be calculated one by one according to the sub-data range to obtain the corresponding calculation pattern results; step S4, using all inference pattern results and calculation pattern results as the data patterns of the sub-data range.

[0014] The data insight device for exploratory data analysis provided by this invention may also have the following features: the data pattern includes salient point_1, salient point_n, dominant point, abrupt change point, outlier, trend, consistency, and periodicity. Salient point_1 represents a sub-data range with the largest aggregate value, satisfying that this value is greater than twice the mean and less than half the sum. Salient point_n represents a sub-data range with the smallest aggregate value, satisfying that this value is less than half the mean. Dominant point represents a sub-data range with the largest aggregate value, satisfying that this value is greater than half the sum of aggregate values. Abrupt change point represents a sub-data range where the value of a group changes significantly compared to the aggregate values ​​of the preceding and following groups. Outlier represents a sub-data range where the aggregate value of a group deviates significantly from the aggregate values ​​of other groups. Trend indicates that the aggregate values ​​of all groups in the sub-data range combine to form increasing or decreasing values. Consistency indicates that the aggregate values ​​of all groups in the sub-data range are very uniform. Periodicity indicates that the sub-data range satisfies the minimum periodicity value of the aggregate values ​​of the groups.

[0015] The data insight device for exploratory data analysis provided by the present invention may also have the following feature: wherein, the commonality analysis is to identify the commonalities and anomalies of the data patterns corresponding to each sub-data range within the same homogeneous data range.

[0016] The role and effect of invention

[0017] According to the data insight device for exploratory data analysis of the present invention, the data pattern of a single-value data range is pre-calculated as a pattern cell by an offline construction module, and data with different aggregation methods under various combinations of dimensional attributes and measurement attributes is used as a data cell. The online analysis module generates data ranges and homogeneous data ranges according to analysis instructions, and uses a pattern cube composed of pattern cells and data cells, along with pruning inference, to prune patterns in each sub-data range, determining data patterns that must exist, data patterns that must not exist, and uncertain data patterns. Uncertain data patterns are then calculated one by one to obtain the data pattern of each sub-data range, and the similarities and differences are analyzed to obtain insight results. Therefore, the data insight device for exploratory data analysis of the present invention can avoid repeated calculations of data range patterns and improve the efficiency of insight discovery. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the working principle of the data insight device in an embodiment of the present invention;

[0019] Figure 2 This is a block diagram of the data insight device in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the data cells and single-value data range in a three-dimensional Cuboid according to an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of various data modes in an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a three-dimensional Cuboid containing pattern cells in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram illustrating the working principle of the data pattern generated by the online analysis module in an embodiment of the present invention;

[0024] Figure 7 This is a flowchart illustrating the data generation mode of the second computing subunit in an embodiment of the present invention;

[0025] Figure 8 This is a schematic diagram of the data insight process in an embodiment of the present invention;

[0026] Figure 9 This is a schematic diagram illustrating the interactive insight discovery efficiency results of various methods in the embodiments of the present invention. Detailed Implementation

[0027] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of the data insight device for exploratory data analysis of the present invention.

[0028] This embodiment provides a data insight device for exploratory data analysis, used to obtain insight results of a target dataset based on user analysis instructions. The data in the target dataset is divided into measurement attributes and dimensional attributes; the measurement attributes are numerical, while the dimensional attributes are non-numerical. The analysis instructions include the user's values ​​for at least one dimensional attribute.

[0029] In this embodiment, a product sales revenue dataset is used as the target dataset to illustrate the data insight device. This product sales revenue dataset includes four attributes: sales revenue, customer gender, age group, and quarter. Sales revenue is a measurement attribute, while the other three attributes are dimensional attributes. Customer gender includes male and female, age group includes youth and senior, and quarter includes 1, 2, 3, and 4.

[0030] Figure 1 This is a schematic diagram illustrating the working principle of the data insight device in an embodiment of the present invention.

[0031] like Figure 1 As shown, in the offline stage, the data insight device aggregates the metadata information of the dataset, namely the dimensional attributes and measurement attributes, to form an initial data cube, namely each data cell. At the same time, it generates single-value data ranges based on the dimensional attributes and measurement attributes, and then obtains pattern cells through data pattern analysis. All pattern cells and data cells are combined to obtain the initial insight cube, namely the pattern cube.

[0032] During the online phase, users input analysis commands through the interactive exploration interface of the data insight device to explore. Based on the analysis commands, data ranges are generated or parsed to obtain single-value data ranges, multi-value data ranges, and drill-down data ranges. Then, a pattern cube is used to perform data pattern analysis on the data ranges, and corresponding homogeneous data ranges are generated. Next, corresponding insights are performed on the homogeneous data ranges, i.e., data pattern analysis is conducted to analyze similarities and differences. Finally, the insight results are obtained and then the insight visualization is generated. The visualized results, i.e., the insight view, are then displayed to the user through the interactive exploration interface.

[0033] Figure 2 This is a block diagram of a data insight device in an embodiment of the present invention.

[0034] like Figure 2 As shown, the data insight device 100 includes an offline construction module 10, an online analysis module 20, and a control module 30 that controls the operation of the above modules.

[0035] The offline construction module 10 is used to construct the corresponding pattern cube based on the target dataset, including a grouping unit 101, an aggregation unit 102, a single-value data range generation unit 103, an offline pattern calculation unit 104, and an offline storage unit 105.

[0036] In this embodiment, the pattern cube PattemCube is composed of multiple i-CuboidSets (1≤i≤n), and its expression is:

[0037] PatternCube={CuboidSet1,...CuboidSet i , ..., CuboidSet n},

[0038]

[0039] In the formula, DimSet ij Cuboid(DimSet) represents the j-th group of the target dataset containing i-th dimension attributes. ij ) contains DimSet ij The aggregated results of all measured attributes after grouping, as well as the pattern analysis results of a portion of the data range, i.e., data cells and pattern cells, CuboidSet i It is composed of all i-dimensional Cuboids.

[0040] The following describes the specific content and construction calculation method of the above-mentioned pattern cube through each unit in the offline construction module 10.

[0041] Grouping unit 101 is used to group all dimensional attributes and combine each grouping result with each measurement attribute to obtain multiple dimensional attribute combinations.

[0042] In this embodiment, the dimensional attribute combinations of the product sales dataset include [customer gender, age group], [customer gender, quarter], [age group, quarter], and [customer gender, age group, quarter].

[0043] The aggregation unit 102 is used to combine attributes of various dimensions and aggregate them according to the corresponding data in the target dataset through various aggregation methods to obtain the corresponding data cells.

[0044] The aggregation methods include "SUM", "MAX", "AVG" and "MIN".

[0045] In this embodiment, SQL statements are used to aggregate the combinations of attributes from various dimensions. Therefore, the SQL statements corresponding to the product sales data set include:

[0046] SELECT SUM(Sales), MIN(Sales), MAX(Sales), AVG(Sales), Customer Gender, Age Range FROM dataset GROUP BY Customer Gender, Age Range.

[0047] SELECT SUM(Sales), MIN(Sales), MAX(Sales), AVG(Sales), Customer Gender, Quarter FROM dataset GROUP BY Customer Gender, Quarter

[0048] SELECT SUM(sales), MIN(sales), MAX(sales), AVG(sales), age group, quarter FROM dataset GROUP BY age group, quarter.

[0049] SELECT SUM(Sales), MIN(Sales), MAX(Sales), AVG(Sales), Customer Gender, Customer Gender, Age Group, Quarter FROM dataset GROUP BY Customer Gender, Age Group, Quarter.

[0050] Among them, Cuboid(DimSet ij The expression for the data cell in () is:

[0051] cell data = <a1,a2,…a i :value>,

[0052] In the formula, cell data Let i be the data cell. data The total number of dimension attributes in the corresponding dimension attribute combination, a i This refers to the specific value of the i-th dimension attribute in this dimension attribute combination, where value is the data cell. data The aggregation result of the measurement attributes in the corresponding dimensional attribute combination after applying the corresponding aggregation method.

[0053] For example, in the product sales dataset, the specific content of Cuboid([[customer gender, age group], SUM(sales amount)]) corresponding to [customer gender, age group] is:

[0054] Customer gender age group SUM (Sales Revenue) male youth 1 million male elderly 900,000 female youth 800,000 female elderly 700,000

[0055] The single-value data range generation unit 103 is used to split and recombine the combination of attributes of each dimension into single values ​​to obtain the corresponding single-value data range.

[0056] For example, in a product sales dataset, a single-value data range can be represented as [A, B, C, D], where A and D are different dimensional attributes with specific values. A represents a data subspace, D represents the subset of data information that the user is interested in, B is the remaining dimensional attribute among the three dimensional attributes, representing the data partitioning attribute, and C is an aggregation method of the measurement attribute, representing the data measurement object. Therefore, with customer gender as 'male' as the data subspace, quarter as 1 as the subset of data information of interest, and sales revenue aggregated using SUM, the single-value data range would be [Customer Gender = 'Male', Age Group, SUM(Sales Revenue), Quarter = 1]. Thus, the product sales dataset can be split and recombined into 192 single-value data ranges of the above type.

[0057] Figure 3 This is a schematic diagram of the data cells and single-value data range in the three-dimensional Cuboid in an embodiment of the present invention.

[0058] like Figure 3 As shown, the dimension attributes corresponding to this data cell are Region, Category, and Segment, and the aggregation operation is Max(discount). The data cell with a dashed circle represents cell = <West, Furniture, Home Office: 0.5>. The data cell with a solid circle represents a single-value data range of <Region: "West", Category, MAX(discount), Segment: "Home Office">, and this data cell can be reused by other single-value data ranges with the same dimension attribute values ​​and aggregation operation, such as <Segment: "Home Office", Category, MAX(discount), Region: "West"> which also corresponds to this set of data.

[0059] In this embodiment, the data cells are constructed using SQL statements, such as the SQL statement "SELECT Region, Category, Segment, MAX(discount) FROM superstoreGROUP BY Region, Category, Segment;". Figure 3 The data cell shown. For example, the single-value data range [Age Group = 'Youth', Quarter, SUM(Sales), Customer Gender = 'Male'], the corresponding SQL statement is "SELECT SUM(Sales), Quarter FROM Dataset WHERE Age Group = 'Youth' AND Customer Gender = 'Male' GROUP BY Quarter".

[0060] The offline mode calculation unit 104 is used to perform data mode calculations on each single-value data range to obtain the corresponding mode cell.

[0061] The expression for the pattern cell is:

[0062] cell pattern = < a1, a2, ... a i :pattern_dict>,

[0063] In the formula, cellpattern is the pattern cell, i is the total number of dimension attributes in the single-value data range corresponding to the pattern cell cellpattern, and a i For the i-th dimension attribute within this single-value data range, pattern_dict represents the pattern cell. pattern The data pattern result is obtained by performing data pattern calculation on the corresponding single-value data range. agg(m) is the aggregation result after agg is applied to the measurement attribute m. The pattern cell has exactly one dimension attribute with the value "ALL", where "ALL" represents all possible values ​​for that dimension attribute.

[0064] In this embodiment, the data patterns include salient point 1, salient point n, dominant point, abrupt change point, outlier, trend, consistency, and periodicity.

[0065] Figure 4 This is a schematic diagram of various data modes in an embodiment of the present invention.

[0066] like Figure 4 As shown, the first column of the table represents the various data patterns, i.e., the pattern types; the second column describes the characteristics of each data pattern, i.e., the highlights of the pattern; and the third column provides examples of the highlights of each pattern, using the monthly sales figures of the Eastern Supermarket over the past six months as the data range.

[0067] Among them, outstanding_1 represents the group with the largest aggregate value in the sub-data range, satisfying the condition that this value is greater than twice the mean and less than half the sum. outstanding_last represents the group with the smallest aggregate value in the sub-data range, satisfying the condition that this value is less than half the mean. The dominant point (attribute) represents the group with the largest aggregate value in the sub-data range, satisfying the condition that this value is greater than half the sum of the aggregate values. The change_point represents a group within the sub-data range whose value changes significantly compared to the corresponding aggregate values ​​of the preceding and following groups. The outlier represents a group within the sub-data range whose aggregate value deviates significantly from the aggregate values ​​of other groups. The trend represents the combination of increasing or decreasing aggregate values ​​across all groups in the sub-data range. Evenness represents the uniformity of aggregate values ​​across all groups in the sub-data range. Seasonality represents the minimum periodicity value for which the aggregate values ​​of the groups in the sub-data range exhibit a periodicity.

[0068] Figure 5 This is a schematic diagram of a three-dimensional Cuboid containing pattern cells in an embodiment of the present invention.

[0069] like Figure 5 As shown, the pattern cell enclosed by a solid circle is represented as cell=<West,ALL,Home Office:{″trend″:″increase″,″outlier″:[″Furniture″]}> This indicates that the data range exhibits an increasing pattern, as well as outliers of the Furniture category.

[0070] Offline storage unit 105 is used to store all data cells and pattern cells as a pattern cube.

[0071] The online analysis module 20 is used to calculate insight results based on analysis instructions and pattern cubes, including a data range generation unit 201, a homogeneous data range generation unit 202, an online pattern calculation unit 203, an insight generation unit 204, a recommendation unit 205, and a visualization unit 206.

[0072] Figure 6 This is a schematic diagram illustrating the working principle of the data pattern generated by the online analysis module in an embodiment of the present invention.

[0073] like Figure 6As shown, the data range is first analyzed to determine whether it includes drill-down data ranges (i.e., data ranges with two or more attributes), single-value data ranges (i.e., single-value data ranges), and multi-value data ranges (i.e., multi-value data ranges). For single-value and multi-value data ranges, it is determined whether there is a corresponding pattern (i.e., a data pattern in the insight cube, i.e., the pattern cube). If not, data is reused from the pattern cube, i.e., the data content corresponding to that data range is retrieved, and then pattern analysis is performed on that data range to obtain the corresponding data pattern. If so, the data pattern is reused from the insight cube, i.e., the corresponding data pattern is retrieved. For single-valued data, the corresponding data pattern is directly used as the result. For multi-valued data, the data patterns corresponding to the single-valued data it contains are used to confirm the sure-existing patterns (sure dp) and the impossible-to-not-existing patterns (impossibledp) through pruning inference. Then, the remaining uncertain patterns (possibledp) are analyzed sequentially according to the data range corresponding to the multi-valued data, thus obtaining the judgment results of all patterns. For drill-down data, it is first expanded into a homogeneous data range, and then each sub-data range is processed according to multi-valued data or single-valued data to obtain the corresponding data pattern.

[0074] The data range generation unit 201 is used to generate at least one corresponding data range according to the analysis instructions and the pattern cube.

[0075] The data range includes both single-valued and multi-valued data ranges. The multi-valued data range is obtained by selecting the corresponding single-valued data range from the pattern cube and performing an aggregation operation.

[0076] For example, if a user is interested in data with gender types including {male, female}, then according to the corresponding analysis instructions, the data range of the product sales dataset can be obtained as follows: Ds1:[Age group = 'youth', quarter, SUM(sales), customer gender = 'male'], Ds2:[Age group = 'youth', quarter, SUM(sales), customer gender = 'female'], Ds3:[Age group = 'youth', quarter, SUM(sales), customer gender in {male, female}], etc.

[0077] The homogeneous data range generation unit 202 is used to generate homogeneous data ranges corresponding to each data range.

[0078] The homogeneous data range contains multiple sub-data ranges. A sub-data range is the data range corresponding to the homogeneous data range, and a data range obtained by expanding the dimensional attribute values ​​of that data range.

[0079] For example, the homogeneous data range obtained by expanding Ds1 is [[Age group = 'Youth', Quarter, SUM (Sales), Customer gender = 'Male'], [Age group = 'Elderly', Quarter, SUM (Sales), Customer gender = 'Male']].

[0080] The online pattern calculation unit 203 is used to calculate the data pattern corresponding to each homogeneous data range according to the pattern cube, including the first calculation subunit 2031 and the second calculation subunit 2032.

[0081] The first calculation subunit 2031 is used to take the data pattern calculation result in the corresponding pattern cell as the data pattern of the sub-data range that belongs to the single-value data range.

[0082] The second calculation subunit 2032 stores pruning inferences, which are used to calculate the data pattern of a sub-data range that belongs to a multi-valued data range based on the relevant single-valued data range and pruning inferences.

[0083] For example, if the data pattern of a single-value data range Ds1 is monotonically increasing with the fourth quarter as the dominant point, and the data pattern of a single-value data range Ds2 is monotonically increasing with the third quarter as the dominant point, then according to the pruning inference, the multi-value data range Ds3 must have a monotonically increasing data pattern and will not have either consistent or periodic data patterns. Therefore, it is necessary to analyze and confirm the five data patterns to be determined, namely "significant point_1", "significant point_n", dominant point, abrupt change point, and outlier point, according to the corresponding data range.

[0084] Figure 7 This is a flowchart illustrating the data generation mode of the second computing subunit in an embodiment of the present invention.

[0085] like Figure 7 As shown, the second computational subunit generates a data pattern by including the following steps:

[0086] Step S1: Based on the calculation results of the data pattern corresponding to the relevant single-value data range and the pruning inference, obtain the inference pattern result.

[0087] Step S2: Remove the inference pattern results from all the patterns to be analyzed to obtain the patterns to be calculated.

[0088] Step S3: Calculate the patterns to be calculated one by one according to the sub-data range to obtain the corresponding calculation pattern results.

[0089] Step S4: Take all inference pattern results and calculation pattern results as data patterns for sub-data ranges.

[0090] The insight generation unit 204 is used to perform commonality analysis on the data patterns of each homogeneous data range to obtain corresponding insights.

[0091] Commonality analysis involves identifying commonalities and anomalies in the data patterns corresponding to various sub-data ranges within the same homogeneous data range. For example, if a homogeneous data range contains four sub-data ranges, and the first three are monotonically increasing while the remaining one is monotonically decreasing, then the commonalities and anomalies of this homogeneous data range can be summarized from these data patterns.

[0092] Recommendation unit 205 is used to perform value analysis on all insights and generate the top-k most valuable insights as the insight results. In this example, top-k = 5.

[0093] The visualization unit 206 is used to visualize the insights and display them to the user.

[0094] The control module 30 stores the control program that controls the operation of each module.

[0095] The following description, in conjunction with the accompanying drawings, illustrates the process of performing data insights using the data insight device 100.

[0096] Figure 8 This is a schematic diagram of the data insight process in an embodiment of the present invention.

[0097] like Figure 8 As shown, data insight using the data insight device 100 includes the following steps:

[0098] Step T1: Group all dimensional attributes using grouping unit 101, and combine each grouping result with each measurement attribute to obtain multiple dimensional attribute combinations.

[0099] Step T2 involves using aggregation unit 102 to combine attributes of each dimension, and then aggregating the corresponding data in the target dataset using various aggregation methods to obtain the corresponding data cells.

[0100] Step T3: The single-value data range generation unit 103 is used to split and recombine the single values ​​of the attribute combinations of each dimension to obtain the corresponding single-value data range.

[0101] Step T4: The offline mode calculation unit 104 is used to perform data pattern calculations on each single-value data range to obtain the corresponding pattern cell.

[0102] Step T5: Use offline storage unit 105 to store all data cells and pattern cells as a pattern cube.

[0103] Step T6: The data range generation unit 201 generates at least one corresponding data range according to the analysis instructions and the pattern cube.

[0104] Step T7: Use homogeneous data range generation unit 202 to generate homogeneous data ranges corresponding to each data range.

[0105] In step T8, the online mode calculation unit 203 calculates the data patterns corresponding to each homogeneous data range based on the mode cube.

[0106] Step T9: The insight generation unit 204 performs commonality analysis on the data patterns of each homogeneous data range to obtain the corresponding insights.

[0107] Step T10: Use recommendation unit 205 to perform value analysis on all insights and generate the top-k most valuable insights as the insight results.

[0108] Step T11: Visualization unit 206 is used to visualize the insight results and display them to the user.

[0109] In this implementation, the workload was tested using the Hotel Booking dataset to evaluate the efficiency of interactive insight discovery using the latest industry-leading automated insight discovery methods, MetaInsight, the pre-computed data cube optimization method Kylin, and the InsightCube method employing the Data Insight Device 100.

[0110] Figure 9 This is a schematic diagram illustrating the interactive insight discovery efficiency results of various methods in the embodiments of the present invention.

[0111] like Figure 9 As shown, the horizontal axis represents different time budgets, and the vertical axis represents the number of insights discovered within the corresponding time budget, i.e., the number of insights. MetaInsight, Kylin, and InsightCube are displayed sequentially from left to right for each time budget. It is evident that, within the same time budget, the InsightCube method calculates an average increase of 184.2% and 154% in the number of insights compared to the MetaInsight and Kylin methods, respectively. This demonstrates that the Data Insight Device 100 significantly improves the efficiency of insight discovery.

[0112] The role and effect of the embodiments

[0113] According to the data insight device for exploratory data analysis involved in this embodiment, the offline construction module pre-calculates data patterns of single-value data ranges as pattern cells, and data with different aggregation methods under various combinations of dimensional attributes and measurement attributes as data cells. The online analysis module generates data ranges and homogeneous data ranges according to analysis instructions. Using a pattern cube composed of pattern cells and data cells, and pruning inference, pattern pruning is performed on each sub-data range to determine data patterns that are certain to exist, data patterns that are certain to not exist, and uncertain data patterns. Uncertain data patterns are then calculated one by one to obtain the data patterns of each sub-data range. Differences and similarities are then analyzed to obtain insight results. In summary, this method avoids repeated calculations of data range patterns, improving the efficiency of insight discovery.

[0114] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A data insight device for exploratory data analysis, used to obtain insight results of a target dataset according to user analysis instructions, characterized in that, include: An offline construction module is used to construct a corresponding pattern cube based on the target dataset; The online analysis module is used to calculate the insight results based on the analysis instructions and the pattern cube. The data in the target dataset is divided into measurement attributes and dimensional attributes. The measurement attribute data is numerical, while the dimension attribute data is non-numerical. The offline build module includes: A grouping unit is used to group all the dimensional attributes and combine each grouping result with each of the measurement attributes to obtain multiple dimensional attribute combinations; An aggregation unit is used to combine the various dimensional attributes and aggregate them according to the corresponding data in the target dataset using various aggregation methods to obtain the corresponding data cells. A single-value data range generation unit is used to perform single-value splitting and recombination on each of the said dimensional attribute combinations to obtain the corresponding single-value data range; The offline mode calculation unit is used to perform data mode calculations on each of the single-value data ranges to obtain the corresponding mode cell. An offline storage unit is used to store all the data cells and the pattern cells as the pattern cube. The online analysis module includes: A data range generation unit is used to generate at least one corresponding data range according to the analysis instructions and the pattern cube; A homogeneous data range generation unit is used to generate homogeneous data ranges corresponding to each of the data ranges; An online pattern calculation unit is used to calculate the data pattern corresponding to each homogeneous data range based on the pattern cube. An insight generation unit is used to perform commonality analysis on the data patterns of each homogeneous data range to obtain corresponding insights. The recommendation unit is used to perform value analysis on all the insights and generate the top-k most valuable insights as the insight results. Specifically, a product sales data set is used as the target dataset. This product sales data set includes four attributes: sales amount, customer gender, age group, and quarter. Sales amount is a measurement attribute, while the other three attributes are dimensional attributes. Customer gender includes male and female, age group includes youth and old age, and quarter includes 1, 2, 3, and 4. The homogeneous data range includes multiple sub-data ranges. The sub-data range refers to the data range corresponding to the homogeneous data range, and the data range obtained by expanding the values ​​of the dimensional attribute of the data range. The online mode calculation unit includes: The first calculation subunit is used to take the data pattern calculation result in the corresponding pattern cell as the data pattern of the sub-data range that belongs to the single-value data range; The second computational subunit stores pruning inferences, used to calculate the data pattern of the sub-data range belonging to the multi-valued data range based on the relevant single-valued data range and the pruning inferences. The data patterns include salient point_1, salient point_n, dominant point, abrupt change point, outlier, trend, consistency, and periodicity. The salient point_1 indicates that the sub-data range has the largest aggregate value, and that the value is greater than twice the mean but less than half the sum of the groups. The salient point_n represents the group within the sub-data range that has the smallest aggregate value, and that value is less than half the mean. The dominant point represents the group whose sub-data range has the largest aggregate value, and that value is greater than half of the sum of the aggregate values. The abrupt change point indicates that the value of a certain group within the sub-data range changes significantly compared to the aggregated value of the corresponding groups before and after it. The outlier indicates that the aggregated value of a certain group within the sub-data range deviates significantly from the aggregated values ​​of other groups. The trend indicates that the aggregated values ​​of all groups within the sub-data range result in either an increasing or decreasing value. The consistency refers to the uniformity of the aggregated values ​​across all groups within the sub-data range. The periodicity refers to the minimum periodicity value of the aggregate value of the group satisfying the condition that the sub-data range exhibits a periodicity.

2. The data insight device for exploratory data analysis according to claim 1, characterized in that: in, The aggregation methods include "SUM", "MAX", "AVG" and "MIN".

3. The data insight device for exploratory data analysis according to claim 1, characterized in that: in, The expression for the data cell is: ， In the formula For the data cell, For data cells The total number of the dimension attributes in the corresponding dimension attribute combination. For the first of the attribute combinations in this dimension The specific values ​​of each dimension attribute For data cells The aggregation result of the measurement attributes in the corresponding dimensional attribute combination after performing the corresponding aggregation method.

4. The data insight device for exploratory data analysis according to claim 1, characterized in that: in, The expression for the pattern cell is: ， In the formula For the pattern cell, For pattern cells The total number of the dimension attributes within the corresponding single-value data range. For the single-value data range, the first The specific values ​​of each dimension attribute For pattern cells The data pattern result obtained by performing data pattern calculation on the corresponding single-value data range. For measuring attributes Aggregation method The final aggregation result The pattern cell has one and only one dimension attribute with a value of "ALL". "ALL" represents all possible values ​​for this dimension attribute.

5. The data insight device for exploratory data analysis according to claim 1, characterized in that: in, The analysis command includes the user's value for at least one of the said dimension attributes. The data range includes both the single-value data range and the multi-value data range containing the given values. The multi-value data range is obtained by selecting the corresponding single-value data range from the pattern cube and performing an aggregation operation.

6. The data insight device for exploratory data analysis according to claim 1, Its features are: The second computing subunit generates the data pattern by including the following steps: Step S1: Based on the calculation results of the data pattern corresponding to the relevant single-value data range and the pruning inference, obtain the inference pattern result; Step S2: Remove the inference pattern result corresponding to the inference pattern from all the patterns to be analyzed to obtain the pattern to be calculated; Step S3: Calculate the pattern to be calculated one by one according to the sub-data range to obtain the corresponding calculation pattern result; Step S4: Take all the inference pattern results and the calculation pattern results as the data pattern of the sub-data range.

7. The data insight device for exploratory data analysis according to claim 1, characterized in that: in, The commonality analysis involves identifying commonalities and anomalies in the data patterns corresponding to each of the sub-data ranges within the same homogeneous data range.

Images