Insight discovery method and system based on subspace encoding and machine learning models

Abstract

The application provides a kind of insight discovery method and system based on subspace coding and machine learning model, and the main application scene is the automatic insight discovery link in exploratory data analysis.The application is directed to the problem that directly using attribute column in data to construct feature vector can cause data range in insight discovery process to be unable to be effectively distinguished, and all feature extraction and coding processes are completed based on the data subspace corresponding to the given data range, to ensure the uniqueness of data range vectorization;For the calculation sequence of multiple data ranges, the priority of data range is predicted using a random forest model, so that the data range most likely to contain insights is mined within the given time budget.The application is more excellent in efficiency and effect compared with other existing insight recommendation methods in exploratory data analysis process, and can mine more valuable data insights in a shorter time and return them to the analyst.

Description

Technical Field

[0001] This invention belongs to the field of data analysis technology, and relates to exploratory data analysis methods and systems in data analysis, specifically to an insight discovery method and system based on subspace encoding and machine learning models. Background Technology

[0002] With the continuous development of communication and information technologies, the rate at which data is generated is increasing rapidly. Simultaneously, with the evolution and development of database and storage technologies, more and more data is being recorded and stored for future development and utilization. Internet search engines support billions of web searches, processing tens of thousands of terabytes of data daily. The ever-increasing volume of data, the deepening applications, and the undeniable value of data compel people to explore how to better benefit from it.

[0003] To help analysts discover the inherent patterns in data, exploratory data analysis (EDA) was proposed. When data analysts face a completely new and unknown dataset, understanding the information and characteristics contained within the dataset during the exploration process to gain basic "insights" becomes particularly important. Currently, there are many excellent exploratory data analysis tools or platforms in the industry to assist users, such as Pandas, Matplotlib, Microsoft Power BI, and Tableau. However, regardless of the tool or platform used, users must manually perform exploratory data analysis. Especially for non-professional data scientists, manually obtaining insights from data requires more "trial and error," which is a tedious and time-consuming process.

[0004] How to automatically and effectively acquire valuable knowledge and data patterns in data exploration has become an increasingly important topic. In recent years, academia and industry have seen a surge of research and applications focused on automatically discovering insights during exploratory data analysis, aiming to address the time-consuming nature of processing multidimensional datasets. Summarizing these methods, we find that the search space involved in automating insight acquisition is often enormous. Current optimization solutions utilize pruning and SQL query performance (shared computation) techniques to improve search efficiency; however, as data volume and dimensionality increase, these optimization methods easily reach performance bottlenecks. For users, if the time cost of automatically acquiring valuable insights is too high, the interaction latency between analysts and users will be significant, which is unacceptable to analysts. Therefore, providing analysts with a fast, lightweight, and accurate method and system for automatically acquiring valuable insights is a crucial problem that urgently needs to be solved. Summary of the Invention

[0005] This invention addresses the aforementioned problems by providing a fast, lightweight, and accurate method for discovering insights from large datasets. The invention employs the following technical solution:

[0006] This invention provides an insight discovery method based on subspace encoding and machine learning models. The method includes an online prediction stage comprising the following steps: generating multiple data ranges based on the dataset to be predicted and user-input filtering conditions; performing subspace-based feature encoding on each data range to obtain feature vectors for each data range; inputting the multiple feature vectors into a trained machine learning model to obtain priority predictions for the corresponding data ranges; and performing insight analysis and recommendation on the highest-priority data ranges to obtain insight analysis and recommendation results. The data range is a triple containing a data subspace, a data partitioning attribute, and a data measurement object. In the subspace-based feature encoding, a label encoding method is used to encode the data subspace, and the label encoding method is used to encode the data partitioning attribute and the data measurement object respectively on the data subspace.

[0007] The insight discovery method based on subspace encoding and machine learning models provided by this invention may also have the following technical features: In the feature encoding based on data subspaces, the label encoding method is used to mark whether the attribute in the data subspace is a categorical attribute or a temporal attribute, and the ratio of the number of records of that attribute in the data subspace to the number of records of that attribute in the dataset; the label encoding method is used to mark whether the data partitioning attribute is a categorical attribute or a temporal attribute, and the number of unique values ​​of that data partitioning attribute in the corresponding data subspace; statistical methods are used to extract the statistical features and distribution features of the data measurement object with numerical attributes in the corresponding data subspace.

[0008] The insight discovery method based on subspace encoding and machine learning model provided by this invention may also have the following technical features, wherein the statistical features include the mean, median, range, variance, standard deviation, 25 / 75 quantile, absolute median deviation, mean absolute deviation, and coefficient of deviation of the data measured in the corresponding data subspace; and the distribution features include the entropy, Gini coefficient, skewness, kurtosis, moments, and normality of the data.

[0009] The insight discovery method based on subspace encoding and machine learning model provided by the present invention may also have the following technical features, wherein, in the priority prediction, the trained machine learning model is first used to obtain the priority prediction score of the data range, and then, based on the priority prediction score, a priority queue is used to construct an ordered queue of all the data ranges in descending order of priority.

[0010] The insight discovery method based on subspace encoding and machine learning models provided by this invention may also have the following technical features: in the insight analysis and recommendation, the data range is expanded into a homogeneous data range according to the order in the priority queue, and it is verified whether there are any insights in the homogeneous data range and the utility of the insights is calculated. The insights obtained within a predetermined time constitute a candidate insight set, and then several of the highest priority insights corresponding to the data range are selected from the candidate insight set for recommendation.

[0011] The insight discovery method based on subspace encoding and machine learning models provided by this invention may also have the following technical features: the method further includes an offline learning stage, which includes the following steps: generating multiple training data ranges based on the training dataset and set filtering conditions; performing the subspace-based feature encoding on each training data range to obtain a feature vector for each training data range; using the feature vectors of the training data ranges as training samples and the corresponding utility scores as labels to train the machine learning model to obtain the trained machine learning model.

[0012] The insight discovery method based on subspace encoding and machine learning model provided by the present invention may also have the following technical features: for the data range where an insight exists, the utility score is the total utility score for calculating the insight for that data range; for the data range where no insight exists, the utility score is set to 0.

[0013] The insight discovery method based on subspace encoding and machine learning model provided by this invention may also have the following technical features, wherein the machine learning model is a random forest model.

[0014] The insight discovery method based on subspace encoding and machine learning models provided by this invention may also have the following technical features: the feature encoding based on data subspace includes a process of mapping the data range to SQL queries, which employs a query caching strategy and an enhanced query strategy; the query caching strategy involves placing the query object and query result into a cache after each SQL query, and then reading the corresponding query result from the cache for the same query object; the enhanced query strategy involves retaining the data subspace and data partitioning attributes of several ordinary SQL queries, and merging their data measurement objects to obtain an enhanced query, which contains SQL statements to be executed for multiple data ranges that have the same data subspace and the same data partitioning attributes.

[0015] This invention provides an insight discovery system based on subspace encoding and machine learning models. The system comprises: a data range generation module for generating multiple data ranges based on a dataset to be analyzed and user input; a subspace encoding module for performing feature encoding on each data range based on the data subspace to obtain feature vectors for each data range; a priority prediction module for inputting the multiple feature vectors into a trained machine learning model to obtain priority predictions for the corresponding data ranges; and an insight analysis and recommendation module for performing insight analysis and recommendation on the highest-priority data ranges to obtain insight analysis and recommendation results. The data range is a triple containing a data subspace, a data partitioning attribute, and a data measurement object. In the feature encoding based on the data subspace, a label encoding method is used to encode the data subspace, and the label encoding method is also used to encode the data partitioning attribute and the data measurement object on the data subspace.

[0016] Invention Function and Effect

[0017] The insight discovery method and system based on subspace encoding and machine learning models according to the present invention includes an online prediction stage. In this stage, multiple data ranges are generated, each data range is feature-encoded based on its data subspace, and priority prediction is performed on the data ranges based on a machine learning model. Then, insight analysis and recommendations are performed on the highest-priority data ranges to obtain insight analysis and recommendation results. Specifically, to address the problem that directly using attribute columns from the dataset to construct feature vectors can lead to ineffective differentiation of data ranges during the insight discovery process, the method of the present invention completes all feature extraction and encoding processes based on the data subspace corresponding to a given data range, ensuring the uniqueness of data range vectorization. To address the issues of large datasets and limited insight analysis time, the method of the present invention uses a machine learning model to prioritize data ranges, thereby enabling insight mining of the data ranges most likely to contain insights within a given time budget, obtaining the most valuable insights.

[0018] As described above, when making exploratory insights on large datasets, the method of this invention can quickly and accurately prune its vast search space, improving the efficiency and quality of insight acquisition. Attached Figure Description

[0019] Figure 1 This is a flowchart of the insight discovery method based on subspace encoding and machine learning model in an embodiment of the present invention;

[0020] Figure 2 This is an example diagram of feature encoding based on data subspace in an embodiment of the present invention;

[0021] Figure 3 This is an interactive schematic diagram of the insight discovery system based on subspace encoding and machine learning model in an embodiment of the present invention;

[0022] Figure 4 This is a comparison chart of the time costs of five insight discovery methods in the comparative examples of this invention;

[0023] Figure 5 This is a comparative chart showing the overall effectiveness of the five insight discovery methods in the comparative examples of this invention. Detailed Implementation

[0024] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following describes in detail the insight discovery method and system based on subspace coding and machine learning model of this invention with reference to embodiments and accompanying drawings.

[0025] <Example>

[0026] This embodiment provides an insight discovery method based on subspace encoding and machine learning models for automatic insight discovery on large datasets.

[0027] Figure 1 This is a flowchart of the insight discovery method based on subspace encoding and machine learning models in this embodiment.

[0028] like Figure 1 As shown, the insight discovery method in this embodiment includes two stages: an offline learning stage and an online prediction stage. The offline learning stage is the process of training a machine learning model used to calculate the priority of data ranges; the online prediction stage is mainly the process of obtaining insight recommendation results when analysts use this method to conduct exploratory data analysis.

[0029] Specifically, the method of this embodiment includes the following steps:

[0030] Step S1: In the offline learning phase, the machine learning model is trained using the training set to obtain a trained machine learning model.

[0031] Step S2, in the online prediction stage, based on the dataset and the exploration conditions input by the user, uses a trained machine learning model to obtain insight discovery results.

[0032] Step S1 specifically includes the following sub-steps:

[0033] Step S1-1: Generate multiple data ranges based on the training dataset and the set filtering conditions.

[0034] Step S1-2: Perform feature encoding based on data subspace for each data range to obtain the feature vector of each data range.

[0035] Steps S1-3 involve using the feature vector as training samples and the corresponding utility scores as training labels to train the machine learning model, thereby obtaining a trained machine learning model.

[0036] Step S2 specifically includes the following sub-steps:

[0037] Step S2-1: Generate multiple data ranges based on the dataset to be analyzed and the filtering conditions input by the user.

[0038] Step S2-2: Perform feature encoding based on data subspace for each data range to obtain the feature vector of each data range.

[0039] Step S2-3: Input the feature vector of each data range into the machine learning model to obtain the corresponding priority prediction.

[0040] Steps S2-4 involve performing insight analysis on several data ranges with the highest priority to obtain insight analysis recommendation results.

[0041] The steps described above will be explained in detail below.

[0042] Step S1-1: Generate multiple data ranges based on the dataset and the set filtering conditions.

[0043] The data scope is a unit automatically generated by the insight discovery system, such as based on the user's input search criteria (filter criteria). This unit is used for mining insights and includes the exploration area and exploration object in the insight discovery process.

[0044] Specifically, the data scope comprises three parts: data subspace, data breakdown attributes, and data measurement objects. These three elements form a triple called the data scope. The data subspace is the subset of data within the dataset that the user is interested in. The data breakdown attributes are the attributes used to perform grouping operations (GROUP BY). The data measurement objects are the actions performed on the attributes to perform aggregation operations.

[0045] Step S1-2: Perform feature encoding based on data subspace for the data range in each dataset to obtain the feature vector of each data range.

[0046] The operation in step S1-2 includes encoding the above three parts.

[0047] In this embodiment, the data range is considered as the unit for each insight discovery, because the data range completely encompasses the explored data region (data subspace) and the explored objects (data partitioning attributes and data measurement objects). Typically, machine learning models only accept numerical objects as input. Clearly, the data range triple cannot be directly used as input to a machine learning model, as it is composed of discrete column information. A common approach is to encode the information contained in the triple using one-hot encoding or word embedding. However, these methods are highly dependent on the column information and semantics of the dataset itself. When the explored dataset changes, existing encoding methods and machine learning models used for a particular dataset become ineffective.

[0048] In steps S1-2, feature encoding based on the data subspace is performed. All feature extraction and encoding processes are completed on the data subspace corresponding to the data range. Specifically, for the triple ds of the data range:=<subspace,breakdown,measure> In this context, the data subspace, data partitioning attribute (breakdown), and data measurement object (measure) each correspond to three attributes in the original dataset, including two dimensional attributes and one measurement attribute. Constructing the feature vector of the data range essentially encodes the features of the three attributes corresponding to the aforementioned triples, thus using a vector composed of the features of each column of the dataset to represent the triples. The attributes corresponding to the data subspace, data partitioning attribute (breakdown), and data measurement object (measure) each play different roles. The data subspace plays two crucial roles: first, it filters out the data subset corresponding to the current data range; second, it expands on its own dimensions to obtain a corresponding homogeneous data range. A homogeneous data range is one whose data partitioning attribute and data measurement object are the same as the corresponding attributes and objects in the original data range, and whose data subspace contains the same attributes as the original data range, but whose attribute values ​​are different from the corresponding attribute values ​​in the original data range.

[0049] During feature encoding, the data subspace is encoded on the original columns of the dataset. In this encoding process, statistical characteristics on that column are calculated. Both data partitioning attributes and data measurement objects operate on the data subspace; that is, all partitioning and measurement occur on subsets of data. Feature encoding for both is performed on the data subspace to avoid the problem of identical encodings for certain data ranges (e.g., multiple data ranges containing the same data partitioning attribute and data measurement object, or multiple data ranges with the same attribute but different attribute values ​​within the data subspace having the same encoding), thus improving the uniqueness and accuracy of the encoding.

[0050] Furthermore, the triples of the data range correspond to two dimensional attributes and one measurement attribute. Dimensional attributes are categorical or temporal attributes, and are typically discrete in the data distribution; while measurement attributes are numerical attributes, and are typically continuous in the data distribution. Therefore, in feature extraction, different types of attributes must employ different encoding methods. In practical applications, numerical attributes often possess more diverse data characteristics, such as variance, Gini coefficient, and skewness.

[0051] In this embodiment, for the encoding of data subspaces, label encoding is used to mark whether the attributes in the data subspace are categorical or temporal attributes, and the ratio of the number of records containing that attribute in the data subspace to the number of records of that attribute in the dataset. For the encoding of data partitioning attributes, label encoding is used to mark whether the data partitioning attribute is categorical or temporal, and the number of unique values ​​of that data partitioning attribute in the corresponding data subspace. For data measurement objects with numerical attributes, statistical methods are used to extract statistical features and distribution features. The statistical features may include the mean, median, variance, etc., of the numerical values ​​of the data measurement objects, and the distribution features may include the entropy, Gini coefficient, skewness, kurtosis, etc.

[0052] Table 1 below lists the attribute features implemented by the method of this embodiment in feature encoding based on data subspace.

[0053] Table 1 List of Attribute Features

[0054]

[0055]

[0056] In Table 1, the first two rows correspond to dimension attributes, and the last two rows correspond to measurement attributes. In this embodiment, the attribute types, namely dimension type and data type, are encoded using a tag encoding method.

[0057] Figure 2 This is an example diagram of feature encoding based on subspace in this embodiment.

[0058] like Figure 2 As shown in this embodiment, the Superstore dataset is used to illustrate subspace-based feature encoding.

[0059] The Superstore dataset is primarily used to analyze the operational status of supermarkets. It includes product information, regional information, category information, and customer group information, totaling 21 attributes. In this embodiment, 15 of these attributes (including 9 category attributes, 2 time attributes, and 4 numerical attributes) are analyzed for insight.

[0060] The data range is generated based on the user-input filtering conditions (SQL statement): "SELECT Segment, AVG(Sales) FROM Superstore WHERE Category = "Technology" GROUP BY Segment;". The data range specifically includes: the data subspace: Category: "Technology", the data partitioning attribute: Segment, and the data measurement object: AVG(Sales). The feature vector of the data range consists of three parts: the encoding results of the data subspace, the data partitioning attribute, and the data measurement object.

[0061] When performing feature encoding on the aforementioned data range, feature encoding is performed on the original columns for the data subspace. In this embodiment, statistical features on the original columns are calculated; the data is divided into attributes and data measurement objects, such as... Figure 1 As shown in the lower right corner of the image, codes 2 and 3 represent the feature encodings for these two features, which are performed in their respective data subspaces.

[0062] Steps S1-3 involve using the feature vector as training samples and the corresponding utility scores as training labels to train the machine learning model, thereby obtaining a trained machine learning model.

[0063] In this process, the machine learning model is used to predict the priority of data ranges. Steps S1-3 generate training corpora for training the model, including training samples and labels. The training samples are the feature vectors of the data ranges obtained in step S1-2, and the corresponding labels are the presence or absence of insight in the data range and the total utility score of the insight, which can be calculated using existing methods. In this embodiment, for data ranges with insight, their total utility score is directly used as the training label. For data ranges without insight, since the priority is distributed between 0 and 1, the training label for data ranges without insight is set to 0. In steps S1-3, the training and storage of the machine learning model are completed based on the training samples and labels generated in the above manner.

[0064] In this embodiment, the machine learning model is a random forest model.

[0065] By following the steps above, a trained machine learning model can be obtained, which can then be used in the subsequent online prediction stage.

[0066] Step S2-1: Generate multiple data ranges based on the dataset and the filtering conditions input by the user. This step is the same as step S1-1 and will not be repeated.

[0067] Step S2-2 involves performing feature encoding based on the data subspace for each data range to obtain the feature vector for each data range. This step is the same as S1-2 and will not be repeated here.

[0068] Step S2-3: Input the feature vector of each data range into the machine learning model to obtain the corresponding priority prediction.

[0069] The process involves using the machine learning model built during the offline learning phase to calculate (predict) priority scores, and then using a priority queue to build an ordered queue of all data ranges from high to low priority.

[0070] In this embodiment, the machine learning model is the random forest model trained through step S1 described above.

[0071] Steps S2-4 involve performing insight analysis and recommendations on several data ranges with the highest priority to obtain the insight analysis and recommendation results.

[0072] The process involves expanding the data range into homogeneous data ranges (a set of multiple data ranges used for mutual reference, differing only in attribute values ​​of data subspaces, with the rest being identical) according to a priority queue, based on the definition of the insight acquisition process. The presence of insights within these ranges is then verified, and the utility of those insights is calculated. Insights acquired within the time budget constitute a candidate insight set, from which the top k (the k highest priority data ranges) insights are selected for recommendation. Finally, a visualization is generated based on the recommended insights and displayed to analysts on the corresponding interactive interface.

[0073] Insight discovery uses data ranges as the unit of computation; performing insight discovery on a dataset is equivalent to mining insights from all data ranges within the dataset. As mentioned above, a data range consists of two dimensional attributes and one measurement attribute. The number of data ranges in a dataset can be enormous. Without considering timeliness, insight discovery methods would need to calculate insights from all data ranges and then recommend the most valuable ones. However, in reality, the interaction time between analysts and the system is limited. Therefore, in this embodiment of the insight discovery method, priority is given to analyzing data ranges that contain the "most valuable" (i.e., those with the highest computational priority) insights, thereby obtaining more valuable insight results within a limited time.

[0074] Furthermore, both the offline learning stage (step S1-2) and the online prediction stage (step S2-2) involve mapping the data range to an SQL query, which is the process of calculating the data in the dataset corresponding to the data range. In order to shorten the time cost of this process, in this embodiment, query caching and enhanced query are also used to optimize the mapping.

[0075] The query caching strategy specifically refers to creating a "query cache" for SQL queries. After each SQL query is completed, the query object and query result are cached. When the same query object is encountered later, the corresponding query result is read from the query cache, thereby reducing the computational overhead of repeated SQL queries.

[0076] Whether prioritizing data ranges or validating patterns in homogeneous data (i.e., verifying whether a data range offers insights and the utility of those insights), the execution process generates a large number of repetitive SQL queries. In steps S1-2, although the encoded data ranges differ each time, the same data subspaces and dimensional attributes still exist within different data ranges, resulting in numerous repetitive SQL queries when calculating their features. Similarly, when expanding data ranges into homogeneous ranges, some identical data ranges may appear in different homogeneous data ranges. Therefore, a query caching strategy can effectively reduce the computational overhead of repetitive SQL queries.

[0077] Enhanced query strategy specifically refers to preserving the data subspace (i.e., filtering conditions) and data partitioning attributes (i.e., GROUP BY objects) of several ordinary SQL queries (i.e., SQL statements executed to retrieve a single data range), and then merging the data measurement objects of these ordinary SQL queries (i.e., forming corresponding aggregation calculations) to obtain an enhanced query. An enhanced query contains the SQL statements executed for multiple data ranges with the same data subspace and the same data partitioning attributes. Although a single augmented query may require more time than a regular query, by combining it with a query caching strategy to cache the results of the augmented query, a large number of subsequent query executions can be avoided, thereby achieving efficiency optimization.

[0078] Table 2 below provides an example of generating an enhanced query based on a normal query.

[0079] Table 2 Examples of Standard and Enhanced Queries

[0080]

[0081] For each round of SQL queries, in addition to the issue of duplicate queries mentioned above, some similar queries will also be generated. During feature encoding based on data subspaces (steps S1-2), if the same data subspace is involved, even if the data partitioning attributes or data measurement objects are different, the corresponding SQL queries will all select the same subset of data and then perform different aggregation calculations. The SQL queries mapped to each data range are similar; if the same data subspace and data partitioning attributes are involved, the same subset of data will be selected, followed by the same data partitioning, and finally, different aggregation calculations will be performed on the same group after partitioning. Therefore, incremental queries can be generated in the above manner.

[0082] This embodiment also provides an insight discovery system for implementing the above method, which includes an exploration interaction interface, a data range generation module, a subspace encoding module, a model training module, a priority prediction module, and an insight analysis and recommendation module.

[0083] The exploration interface allows users to input exploration conditions (filtering conditions) and displays corresponding insights and findings to the user.

[0084] The data range generation module is used to implement the above steps S1-1 and S2-1.

[0085] The subspace encoding module is used to implement steps S1-2 and S2-2 above.

[0086] The model training module is used to implement steps S1-3 above.

[0087] The priority prediction module is used to implement steps S2-3 above.

[0088] The insight analysis and recommendation module is used to implement the above steps S2-4 and generate corresponding visualization results based on the insight findings obtained in step S2-4.

[0089] Figure 3 This is an interactive schematic diagram of the insight discovery system based on subspace encoding and machine learning models in this embodiment.

[0090] like Figure 3As shown, during use, the user inputs the dataset to be explored and its attribute information (such as column types) into the system through the exploration interface. The data range generation module automatically generates a data range queue based on the user input. The priority prediction module uses an offline-built random forest model to predict the priority of multiple data ranges and stores the data ranges in the priority queue. Finally, the insight analysis module mines the insights existing in the data ranges in priority order within a limited time and recommends and visualizes the top k (top-k) insights. The system then returns the results to the user through the exploration interface.

[0091] The role and effect of the embodiments

[0092] The insight discovery method and system based on subspace encoding and machine learning models provided in this embodiment include an online prediction stage. In this stage, multiple data ranges are generated, each data range undergoes feature encoding based on its data subspace, and priority prediction is performed on the data ranges based on a machine learning model. Then, insight analysis and recommendations are performed on the highest-priority data ranges to obtain the insight analysis and recommendation results. Specifically, to address the problem that directly using attribute columns from the dataset to construct feature vectors can lead to ineffective differentiation of data ranges during the insight discovery process, the method in this embodiment completes all feature extraction and encoding processes based on the data subspace corresponding to a given data range, ensuring the uniqueness of the data range vectorization. To address the issues of large datasets and limited insight analysis time, the method in this embodiment uses a machine learning model to prioritize data ranges, thereby enabling insight mining of the data ranges most likely to contain insights within a given time budget, yielding the most valuable insights.

[0093] As described above, when making exploratory insights on large datasets, the method in this embodiment can quickly and accurately prune its vast search space, improving the efficiency and quality of insight acquisition.

[0094] Furthermore, the method in this embodiment also employs two strategies, query caching and enhanced query, to accelerate the mapping from data range to SQL query. Query caching can avoid duplicate SQL queries, thereby reducing time overhead. By combining enhanced query and query caching, a large number of subsequent homogeneous query executions can be avoided, further optimizing efficiency and thus improving the efficiency of insight acquisition.

[0095] <Comparative Example>

[0096] This comparative example provides four insight discovery methods to be compared with the insight discovery methods of the embodiments to illustrate the effectiveness of the insight discovery methods of the embodiments. For ease of description below, the four methods used for comparison are referred to as the first method to the fourth method.

[0097] The first method is the MetaInsight method in the prior art, which uses a "first-in, first-out (FIFO)" approach to prioritize data ranges.

[0098] The second method is the MetaInsight method in the prior art, which uses the "number of records involved in the data subspace (count(*))" to prioritize the data range.

[0099] The third method is a variation of the method described in the embodiments, the only difference being that the machine learning model used therein is a multilayer perceptron neural network (MLP) model.

[0100] The fourth method is also a variation of the method described in the embodiments, the only difference being that the machine learning model used therein is a linear regression (LR) model.

[0101] Given that the existing method MetaInsight in the field provides a unified definition of the behavioral paradigm of the current main insight discovery process, and is relatively representative, MetaInsight is used as the main object for evaluation and comparison in terms of efficiency and utility in this comparative example. The first and second methods described above can be used to verify whether the method of the embodiment is superior to the method in the prior art.

[0102] Furthermore, the third and fourth methods described above can be used to verify whether the random forest model used in the embodiments is the most suitable.

[0103] In this comparative example, three datasets were selected for the training of machine learning models to generate training corpora and train and build models in the offline learning phase, including a large dataset Hotel Booking and two relatively small datasets Car Sales and Toy Dataset.

[0104] The Hotel Booking dataset contains booking information for both city hotels and resort hotels. Information includes booking creation time, length of stay, number of adults, children, and infants, number of parking spaces required, and number of special requests. The dataset contains nearly 120,000 data tuples with 32 attributes. This comparative analysis focuses on 27 of these attributes (11 categorical, 4 temporal, and 12 numerical attributes).

[0105] Figure 4 This is a time cost comparison chart of the five insight discovery methods in this embodiment, showing the time cost of each of the five different methods to achieve "the same recommendation utility".

[0106] Figure 5 This is a comparison chart of the total utility of the five insight discovery methods in this embodiment, showing the bar chart of the "recommendation utility score" obtained by the five different methods within a given five-second analysis time.

[0107] like Figure 4 and Figure 5 As shown, it can be intuitively seen that the method of the embodiment performs better in insight recommendation on the above three datasets, which shows that the method of the embodiment can effectively deal with new datasets and recommend valuable insights.

[0108] Furthermore, the comparison of the bar charts clearly shows that, regardless of the time taken for the same recommended utility (same effect) or the recommended utility taken for the same time, the insight discovery method in this example is significantly better than the other four comparative methods.

[0109] Furthermore, the comparison of the bar charts shows that, regardless of the time taken to achieve the same effect or the effect achieved with the same time, the method using the random forest model is significantly better than the methods using the MLP model and the LR model, which also verifies that the random forest model is the most suitable one in the embodiments.

[0110] The above embodiments are merely illustrative of specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are only for illustrating the principles of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An insight discovery method based on subspace encoding and machine learning models, characterized in that, include: The online prediction phase includes the following steps: Multiple data ranges are generated based on the dataset to be predicted and the filtering conditions input by the user. Each data range is subjected to feature encoding based on data subspace to obtain a feature vector for each data range. The multiple feature vectors are respectively input into the trained machine learning model to obtain the priority prediction of the corresponding data range; Insight analysis and recommendations are performed on the highest priority data ranges to obtain insight analysis and recommendation results. The data range refers to a triple that includes a data subspace, data partitioning attributes, and data measurement objects. In the feature encoding based on data subspace, a label encoding method is used to encode the data subspace, and the label encoding method is also used to encode the data partitioning attributes and the data measurement objects in the data subspace. In the feature encoding based on data subspaces, the label encoding method is used to mark whether the attribute in the data subspace is a categorical attribute or a temporal attribute, and the ratio of the number of records of that attribute in the data subspace to the number of records of that attribute in the dataset is also used. The aforementioned label encoding method is used to mark whether the data partitioning attribute is a categorical attribute or a temporal attribute, and the number of unique values ​​of the data partitioning attribute in the corresponding data subspace. Statistical methods are used to extract the statistical and distributional characteristics of the data measurement object with numerical attributes in the corresponding data subspace.

2. The insight discovery method based on subspace encoding and machine learning models according to claim 1, characterized in that: in, The statistical characteristics include the mean, median, range, variance, standard deviation, 25 / 75 quantile, absolute median, mean absolute deviation, and coefficient of variation of the data measured by the data object in the corresponding data subspace. The distribution characteristics include the entropy, Gini coefficient, skewness, kurtosis, moments, and normality of the data.

3. The insight discovery method based on subspace encoding and machine learning models according to claim 1, characterized in that: in, In the priority prediction, the trained machine learning model is first used to obtain the priority prediction score of the data range, and then a priority queue is used to construct an ordered queue of all the data ranges in descending order of priority based on the priority prediction score.

4. The insight discovery method based on subspace encoding and machine learning models according to claim 3, characterized in that: in, In the insight analysis and recommendation, the data range is expanded into a homogeneous data range according to the order in the priority queue, and it is verified whether there are any insights in the homogeneous data range and the utility of the insights is calculated. The insights obtained within a predetermined time constitute a candidate insight set, and then several of the highest priority insights corresponding to the data range are selected from the candidate insight set for recommendation.

5. The insight discovery method based on subspace encoding and machine learning models according to claim 1, characterized in that, Also includes: The offline learning phase includes the following steps: Multiple training data ranges are generated based on the training dataset and the set filtering conditions; The feature encoding based on data subspace is performed on each of the training data ranges to obtain the feature vector of each of the training data ranges; The feature vectors of the training data range are used as training samples, and the corresponding utility scores are used as labels to train the machine learning model, thereby obtaining the trained machine learning model.

6. The insight discovery method based on subspace encoding and machine learning models according to claim 5, characterized in that: in, For the data range where insights exist, the utility score is the total utility score used to calculate the insights for that data range. For the data range where no insight exists, the utility score is set to 0.

7. The insight discovery method based on subspace encoding and machine learning models according to any one of claims 1-6, characterized in that: in, The machine learning model is a random forest model.

8. The insight discovery method based on subspace encoding and machine learning models according to any one of claims 1-6, characterized in that: in, The feature encoding based on data subspaces includes a process of mapping the data range to SQL queries, which employs query caching strategies and enhanced query strategies. The query caching strategy is as follows: after each SQL query, its query object and query result are placed in the cache; subsequently, for the same query object, the corresponding query result is read from the cache. The enhanced query strategy is to retain the data subspace and the data partitioning attribute for several ordinary SQL queries, and merge the data measurement objects to obtain an enhanced query. The enhanced query contains SQL statements to be executed for multiple data ranges that have the same data subspace and the same data partitioning attribute.

9. An insight discovery system based on subspace encoding and machine learning models, characterized in that, include: The data range generation module is used to generate multiple data ranges based on the dataset to be predicted and the filtering conditions input by the user. The subspace encoding module is used to perform feature encoding based on the data subspace for each of the data ranges to obtain the feature vector of each of the data ranges. The priority prediction module is used to input multiple feature vectors into the trained machine learning model to obtain priority predictions for the corresponding data ranges. as well as The insight analysis and recommendation module is used to perform insight analysis and recommendations on several data ranges with the highest priority, and to obtain insight analysis and recommendation results. The data range refers to a triple that includes a data subspace, data partitioning attributes, and data measurement objects. In the feature encoding based on data subspace, a label encoding method is used to encode the data subspace, and the label encoding method is also used to encode the data partitioning attributes and the data measurement objects in the data subspace. In the feature encoding based on data subspaces, the label encoding method is used to mark whether the attribute in the data subspace is a categorical attribute or a temporal attribute, and the ratio of the number of records of that attribute in the data subspace to the number of records of that attribute in the dataset is also used. The aforementioned label encoding method is used to mark whether the data partitioning attribute is a categorical attribute or a temporal attribute, and the number of unique values ​​of the data partitioning attribute in the corresponding data subspace. Statistical methods are used to extract the statistical and distributional characteristics of the data measurement object with numerical attributes in the corresponding data subspace.

Images