Associated data identification alarm method and device based on data visualization

Through data visualization technology and association learning algorithms, data abnormalities are identified and warned, and the problem of data analysis in the existing technology cannot be related to real-time, and the timeliness and efficiency of enterprise decision-making is improved.

CN119066109APending Publication Date: 2024-12-03紫金诚征信有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411091546.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing data analysis technologies cannot realize real-time data correlation analysis and abnormal warning, resulting in delays in corporate decision-making and unable to meet the needs of rapidly adjusting market strategies.

Method used

Using the associated data identification and alarm method based on data visualization, the data is received by receiving the data input by the user, the dimension data and measurement data are determined, the project relationship mode table is constructed using the association learning algorithm, and the data characteristics are input into the isolated forest algorithm model for abnormal evaluation, and the output abnormality is output. Correlate data features to the visual interface to alert.

Benefits of technology

Real-time data relevance is realized, timeliness and efficiency of data analysis is improved, and it can quickly respond to enterprise decision-making needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066109A_ABST
    Figure CN119066109A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an associated data identification alarm method and device based on data visualization, and the method comprises the steps: receiving data inputted by a user, determining corresponding dimension data and measurement data, determining corresponding data basic characteristics according to the dimension data, the measurement data and a statistical analysis rule, and carrying out the identification of the associated data according to the data basic characteristics. Inputting the data features into an association learning algorithm, determining a corresponding project relationship mode table, extracting associated data features in the project relationship mode table, inputting the associated data features into a preset isolated forest algorithm model to determine a corresponding decision forest, and performing anomaly evaluation operation on decision trees in the decision forest respectively. And performing integration operation on the exception score obtained through the exception evaluation operation, determining exception associated data features according to the integration score and a preset exception threshold value, and outputting the exception associated data features to a visual interface for alarm display, so that real-time prediction alarm can be performed based on data relevance identification, and the data analysis timeliness is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method and device for identifying and warning associated data based on data visualization. Background Art

[0002] Data analysis refers to the process of using appropriate statistical analysis methods to analyze large amounts of collected data, extract useful information and form conclusions to conduct detailed research and summarize the data. With the advent of the data age, most functions in business intelligence, risk management, product development, marketing, medical research, and social sciences need to rely on data analysis. By analyzing the correlation between data, corporate strategies can be adjusted to increase corporate profits.

[0003] At present, data analysis is mainly done by data analysts. Through data analysts' understanding of corporate logic and mastery of data analysis skills, data analysis is performed and visualized. By analyzing data correlation, the company's market strategy can be effectively adjusted based on the correlated data. However, in today's era, companies need to quickly obtain correlated data to adjust corporate decisions to ensure that the company leads the market.

[0004] However, the work of data analysts requires redefining integration standards for different enterprises, the cost of functional maintenance is high, single work is time-consuming and has poor reusability, and the results of enterprise data correlation analysis cannot be presented in real time, which cannot meet the requirements of enterprises for efficient and timely data correlation analysis. A method that can automatically perform data correlation analysis and issue early warnings for correlation anomalies is urgently needed to solve the above problems. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a method and device for identifying and warning related data based on data visualization, which can perform real-time prediction and warning based on data correlation identification, thereby improving the timeliness of data analysis.

[0006] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a method for identifying and warning associated data based on data visualization, comprising:

[0008] Receive data input by a user, perform a data detection operation on the data input by the user to determine a corresponding data type, perform a data recognition operation according to the data type to determine corresponding dimension data and metric data;

[0009] Determine the corresponding basic data characteristics according to the dimension data, the measurement data and the preset statistical analysis rules, input the data features in the basic data characteristics into the association learning algorithm, determine the corresponding data association rules, and construct the corresponding project relationship pattern table according to the data association rules;

[0010] Extract the associated data features in the project relationship pattern table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and a preset anomaly threshold, and output the abnormal associated data features to a visualization interface for alarm display.

[0011] Further, the receiving of data input by a user, performing a data detection operation on the data input by the user, and determining a corresponding data type includes:

[0012] Receiving data input by a user, determining whether the data input by the user conforms to a preset data type detection rule, and if so, determining a corresponding first data type according to the preset data type detection rule;

[0013] If not, determining the corresponding second data type according to the data input by the user and the type inference function;

[0014] According to the first data type and the second data type, a corresponding data type is determined.

[0015] Further, performing a data identification operation according to the data type to determine corresponding dimension data and metric data includes:

[0016] Extracting the field name in the data type, performing field mapping operation according to the field name and a preset business field list, and determining corresponding dimension features and metric features;

[0017] Corresponding dimensional data and metric data are determined according to the dimensional features, the metric features and a preset business rule engine.

[0018] Furthermore, the determining of the corresponding basic characteristics of the data according to the dimensional data, the metric data and the preset statistical analysis rules includes:

[0019] Performing statistical analysis operations and visual analysis operations on the dimension data and the metric data to respectively determine corresponding data center trends and data distribution trends;

[0020] Corresponding basic data characteristics are determined according to the data center trend and the data distribution trend.

[0021] Furthermore, the step of inputting the associated data features into a feature selection layer of a preset isolation forest algorithm model to perform a feature selection operation and determine corresponding selection features includes:

[0022] Inputting the associated data features into a feature selection layer of a preset isolation forest algorithm model to perform a random feature selection operation to determine corresponding random features;

[0023] Performing a random segmentation point selection operation on the random feature to determine a corresponding random segmentation point;

[0024] A corresponding selection feature is determined according to the random feature and the random segmentation point.

[0025] Furthermore, the step of inputting the selected features into the decision tree construction layer of the preset isolation forest algorithm model, performing a decision forest construction operation according to a preset decision tree quantity parameter, and determining a corresponding decision forest includes:

[0026] Input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision tree construction operation according to a preset recursive rule and a preset depth parameter, and determine a corresponding decision tree;

[0027] A corresponding decision forest is determined according to the decision tree and a preset decision tree quantity parameter.

[0028] Furthermore, performing abnormal evaluation operations on the decision trees in the decision forest respectively includes:

[0029] Calculating the path lengths of data points of the decision trees in the decision forest respectively to determine the corresponding first path lengths;

[0030] Calculating the average path lengths of the decision trees in the decision forest respectively to determine the corresponding second path lengths;

[0031] A corresponding anomaly score is determined according to the first path length and the second path length.

[0032] Furthermore, the anomaly scores obtained through the anomaly evaluation operation are integrated to determine the corresponding integrated scores, and the anomaly-related data features output by the preset isolation forest algorithm model are determined according to the integrated scores and a preset anomaly threshold, including:

[0033] Performing a weighted average integration operation on the anomaly scores obtained through the anomaly evaluation operation to obtain a corresponding integration score;

[0034] It is determined whether the integration score exceeds a preset abnormal threshold, and if so, the associated data feature corresponding to the integration score is output as an abnormal associated data feature.

[0035] In a second aspect, the present application provides a device for identifying and warning related data based on data visualization, comprising:

[0036] A data identification module is used to receive data input by a user, perform a data detection operation on the data input by the user, determine a corresponding data type, perform a data identification operation according to the data type, and determine corresponding dimension data and metric data;

[0037] An association rule determination module is used to determine the corresponding basic characteristics of the data according to the dimension data, the measurement data and the preset statistical analysis rules, input the data features in the basic characteristics of the data into the association learning algorithm, determine the corresponding data association rules, and construct the corresponding project relationship pattern table according to the data association rules;

[0038] An associated anomaly alarm module is used to extract associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and a preset anomaly threshold, and output the abnormal associated data features to a visualization interface for alarm display.

[0039] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for identifying and warning associated data based on data visualization when executing the program.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for identifying and warning associated data based on data visualization.

[0041] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for identifying and warning associated data based on data visualization.

[0042] It can be seen from the above technical scheme that the present application provides a method and device for identifying and warning related data based on data visualization, which receives data input by the user, determines the corresponding dimensional data and metric data, determines the corresponding basic characteristics of the data according to the dimensional data, metric data and statistical analysis rules, inputs the data features into the association learning algorithm, determines the corresponding project relationship pattern table, extracts the related data features in the project relationship pattern table, inputs the related data features into the preset isolation forest algorithm model to determine the corresponding decision forest, performs abnormal evaluation operations on the decision trees in the decision forest respectively, performs integration operations on the abnormal scores obtained through the abnormal evaluation operations, determines the abnormal related data features according to the integrated scores and the preset abnormal thresholds, and outputs the abnormal related data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data correlation identification, thereby improving the timeliness of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is one of the flow charts of the method for identifying and warning associated data based on data visualization in an embodiment of the present application;

[0045] Figure 2 This is a second flow chart of the method for identifying and warning associated data based on data visualization in an embodiment of the present application;

[0046] Figure 3 This is a flowchart of the method for identifying and warning related data based on data visualization in an embodiment of the present application;

[0047] Figure 4 This is a fourth flow chart of the method for identifying and warning associated data based on data visualization in an embodiment of the present application;

[0048] Figure 5 This is a fifth flow chart of the method for identifying and warning associated data based on data visualization in an embodiment of the present application;

[0049] Figure 6This is a sixth flow chart of the method for identifying and warning associated data based on data visualization in an embodiment of the present application;

[0050] Figure 7 This is a flowchart of the method for identifying and warning related data based on data visualization in an embodiment of the present application.

[0051] Figure 8 This is a flowchart of the method for identifying and warning associated data based on data visualization in an embodiment of the present application.

[0052] Fig. 9 This is a structural diagram of a correlation data identification and alarm device based on data visualization in an embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0054] Reference numerals:

[0055] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0058] Considering that data analysis is currently mainly done by data analysts, data analysts need to redefine integration standards for different enterprises, the function maintenance cost is high, the single task is time-consuming and has poor reusability, and the enterprise data association analysis results cannot be presented in real time, which cannot meet the enterprise's requirements for efficient and timely data association analysis. The present application provides a method and device for identifying and warning related data based on data visualization, which receives data input by a user, determines the corresponding dimension data and metric data, determines the corresponding basic characteristics of the data according to the dimension data, metric data and statistical analysis rules, inputs the data features into the association learning algorithm, determines the corresponding project relationship pattern table, extracts the related data features in the project relationship pattern table, inputs the related data features into the preset isolation forest algorithm model to determine the corresponding decision forest, performs abnormal evaluation operations on the decision trees in the decision forest respectively, performs integration operations on the abnormal scores obtained through the abnormal evaluation operations, determines the abnormal related data features according to the integrated scores and the preset abnormal thresholds, and outputs the abnormal related data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data association identification, and improving the timeliness of data analysis.

[0059] In order to perform real-time prediction and warning based on data correlation identification and improve the timeliness of data analysis, the present application provides an embodiment of a method for identifying and warning related data based on data visualization, see Figure 1 The method for identifying and warning related data based on data visualization specifically includes the following contents:

[0060] Step S101: receiving data input by a user, performing a data detection operation on the data input by the user to determine a corresponding data type, performing a data recognition operation according to the data type to determine corresponding dimension data and metric data;

[0061] Optionally, in the present embodiment, the purpose of this step is to clean and standardize the data input by enterprise users, to automatically determine the data of different departments and with different definitions of users as the same category, and to identify the data according to the data category, to accurately classify the data into two major categories, namely dimensional data and metric data, to unify the dimension and scale of the metric data, and to lay a solid data processing foundation for the subsequent accurate analysis of the correlation between the data.

[0062] Optionally, in this embodiment, the main technical method of this step is, after receiving the data input by the user, first, use the ETL (Extract, Transform, Load) tool to extract data from different data sources, automatically identify and process missing values, abnormal values ​​and data format standardization, and use the average value to fill in the missing values, so as to maximize the accuracy of the data. Secondly, after the above data is extracted and preprocessed, the preprocessed data is detected by the data detection algorithm to obtain the data type after detection. The data type includes: string, integer, floating point number, date, etc. Finally, the dimension data and metric data are obtained according to the data characteristics of the data type (such as field name), the preset field list and the preset business rule engine.

[0063] Optionally, in this embodiment, an order data analysis of an online retail platform is used as an example:

[0064] Data is collected from order management systems, inventory systems, user behavior tracking systems, etc. Since the data definitions used by the three departments are different, the first step is to preprocess the data, i.e. clean the data, including processing missing values, outliers, and data format standardization. Secondly, NLP technology is used to analyze the field names, and the data detection algorithm determines the data type of the field (such as integer, floating point number, string, date and time, etc.) for preliminary classification. For example, the field name "order number" or "order label" may be identified as the "integer" type, and the field name "start date" or "start date" may be identified as the "date and time" type. Finally, according to the business rules and data type recognition results, the data fields are automatically classified into dimension data and measurement data. For example, if the field is of date and time type, it may be dimension data; if the field is of numeric type, it may be measurement data. Further classification is performed based on the field name or content using regular expressions or keyword matching.

[0065] Optionally, in this embodiment, the data detection algorithm in the above method is completed in two steps.

[0066] First, determine whether the data characteristics entered by the user belong to the preset data type detection rules. For example, if the existing standard data type detection rules: the field name "Order Number" is the "Integer" type, then if the field name entered by the user is "Order Number", the data type is determined according to the rules.

[0067] Secondly, if the data characteristics of the user input do not conform to the rules, the type inference function is used to infer the type. For example, if the field name entered by the user is "order label" and does not conform to the existing standard data type detection rules, then it is inferred as "integer" type according to the type inference function.

[0068] Optionally, in the present embodiment, in the above data detection algorithm, the inference function is to use a special type inference function, which attempts to convert the data into different types and infers the most likely type based on success or failure. For example, if the field name entered by the user is "order label", which does not conform to the existing standard data type detection rules, the inference function uses a set of predefined rules to identify the type of the data field. For example, if the field value contains only numbers, it may be inferred to be an integer or floating point type.

[0069] Optionally, in this embodiment, in the above-mentioned data detection algorithm, the inference function can be replaced by a machine learning algorithm based on demand, and the machine learning algorithm is applied to learn the type of data fields, which involves training models to recognize different types of data patterns, and then using these models to infer the type of new data.

[0070] Optionally, in this embodiment, on the one hand, the best way to infer the data type is rule-based type inference, because most of the data input by the user is structured tabular data, and the rule-based method is effective enough, and does not require model training, saving time. On the other hand, if the data input by the user is disorganized, a machine learning algorithm is suitable for this case. Using a machine model to learn the data model in the user data saves time compared to manually defining rules. In general, based on the above method, before performing data detection, features that are helpful for type inference, such as numerical range, format, length, etc., are first extracted from the data, and the extracted features are compared with known data type patterns. Based on the matching results and possible rules or models, the most likely type of the data field is determined.

[0071] It can be understood that the above steps are to determine the data type. Correct data type determination can ensure the accuracy of the data and lay a solid foundation for subsequent data feature identification.

[0072] Optionally, in this embodiment, determining the dimension data and the measurement data in the above method is implemented by the following steps:

[0073] 1. Dimensional feature identification:

[0074] Dimensional features are usually used in the context of classification, grouping or analysis, such as time, location, product category, etc.

[0075] Automatically identify dimension features by matching keywords in field names with business rules.

[0076] For example, a field containing "ID", "Name", or "Email" might be considered a dimension.

[0077] 2. Metric feature recognition:

[0078] Metric characteristics are quantifiable values ​​used to measure or evaluate, such as sales, profit, volume, etc.

[0079] Automatically identify metric features by matching keywords in field names with business rules.

[0080] For example, a field containing terms such as "sales", "quantity", etc. might be identified as a measure.

[0081] 3. Business rule engine application:

[0082] Use the preset business rules engine to further refine field classification and ensure data accuracy and applicability. The engine can apply the above rules to automatically classify data fields as dimensional data or metric data.

[0083] 4. Determination of dimensional data and measurement data:

[0084] Based on the field mapping results and the output of the business rule engine, it is ultimately determined which fields belong to dimensional data and which belong to metric data.

[0085] It should be noted that for measurement data, its dimension needs to be determined (such as meter, kilogram, RMB, etc.). Machine learning models are usually used to identify and convert measurement data of different dimensions to a unified scale.

[0086] For example, let's say a large retail chain wants to analyze its sales data:

[0087] Extract fields: product number, product name, category, sales date, sales quantity, unit price

[0088] Feature Identification: "Category" and "Sales Date" are identified as dimension features, and "Sales Quantity" and "Unit Price" are identified as measure features.

[0089] Apply the business rule engine: Determine that the "Sales Date" field needs to be further subdivided into dimensions by year, quarter, and month.

[0090] Data determination: According to mapping and business rules, "category" and "sales date" (broken down into year, quarter, and month) are classified as dimension data; "sales quantity" and "unit price" are classified as measure data.

[0091] It can be understood that the above steps are a process of identifying and classifying data. The data defined by the user according to different rules can be re-identified and classified according to its data characteristics. The dimensional data and metric data obtained in this way have unified data characteristics, laying the foundation for the subsequent discovery of data correlation based on data characteristics.

[0092] Step S102: determining corresponding basic data characteristics according to the dimension data, the measurement data and preset statistical analysis rules, inputting data features in the basic data characteristics into an association learning algorithm, determining corresponding data association rules, and constructing a corresponding project relationship pattern table according to the data association rules;

[0093] Optionally, in this embodiment, the purpose of this step is to perform data analysis based on the data with uniform characteristics obtained in step S101 to find the correlation therein so as to improve the operating efficiency of the enterprise.

[0094] Optionally, in this embodiment, the main technical solution for this step is to perform data analysis on dimensional data and metric data to obtain data trends, determine basic data characteristics based on data trends, extract data features therein, use an association learning algorithm to discover the correlation between data features, and construct a corresponding project relationship pattern table.

[0095] Optionally, in this embodiment, for example, suppose there is sales data of a supermarket, and the data set contains multiple transactions, each transaction lists the goods purchased by the customer, and you want to find the correlation between the purchased goods to determine which goods are often purchased together. First, through step S101, determine the data classification with unified data characteristics: dimensional data and metric data. Then, through step S102, get the correlation in the data: retailers find that customers often buy bread while buying milk. In this way, correlation data analysis can help retailers better understand customers' purchasing behavior. At the same time, retailers can greatly improve corporate sales efficiency and profits by placing milk and bread together.

[0096] Optionally, in this embodiment, first, descriptive statistical analysis is performed on the measurement data, including calculating the mean, median, standard deviation, minimum and maximum values, etc., to understand the central tendency and dispersion of the data. For dimensional data, the trend under different dimensions can be identified by grouping the mean and sum of the measurement data. Through statistical analysis, the central tendency of the data can be obtained. Among them, the statistical analysis uses descriptive statistical analysis, which can be achieved by a preset statistical analysis algorithm.

[0097] Secondly, use bar charts, line charts, scatter plots and other charts to show the distribution and trend of metric data. Use box plots to show the distribution of data, including median, quartiles and outliers. Use heat maps or tree diagrams to show the relationship between data of different dimensions. Through visual analysis, the distribution trend of data can be obtained. Visual analysis can be performed through libraries such as Matplotlib, Seaborn or Plotly for data visualization.

[0098] Optionally, in this embodiment, an example is given:

[0099] Dimensional data: time (year, month), region (city), product type

[0100] Metrics: Sales, Sales Quantity

[0101] Statistical analysis: Calculate the mean, median, standard deviation, minimum, and maximum of sales and sales volume. Determine the central tendency and dispersion of the data. Draw a line graph for time series data (such as monthly or annual sales data) to observe the trend of sales and sales volume over time. Analyze sales and sales volume by region and product type, using grouping statistics.

[0102] Visual analysis: Use bar charts to show sales and sales quantities by region or product type. Use box plots to show the distribution of sales and sales quantities and identify outliers. Use scatter plots to analyze the relationship between sales and sales quantities and check whether there is a linear relationship.

[0103] It is understandable that through the above analysis, the trends between the various data are obtained, and the basic characteristics between the products can be obtained by analyzing the trends, for example, determining which products have the characteristics of high sales and frequent purchases. These characteristics lay a solid foundation for the subsequent extraction of characteristic features and the discovery of the correlation relationship therein.

[0104] Optionally, in this embodiment, based on the results of trend analysis, basic characteristics of the data are determined, such as stability, periodicity, outliers, etc. Useful data features are extracted from the basic characteristics of the data, which are meaningful for association rule learning, such as: the frequency of item occurrence, the co-occurrence frequency between items, the volatility or stability index of the data.

[0105] It can be understood that through the above steps, a data feature containing data trends is obtained, which lays the foundation for the subsequent discovery of the correlation between data through association algorithms.

[0106] Optionally, in this embodiment, the association learning algorithm is a frequent pattern growth model (FP-Growth), which is a tree-based data structure to compress the database and discover frequent item sets through a hierarchical structure. Frequent item sets refer to setting a support threshold. If the support of an item set reaches (is greater than or equal to) the threshold, the item set becomes a frequent item set. The corresponding association rules can be found through frequent item sets.

[0107] For example, take a large fresh food supermarket as an example:

[0108] First, association rules can be obtained based on the frequent item sets in the above steps. Take a supermarket data set as an example (the data set has been processed in step S101 and has uniform data features):

[0109] Customer Name Purchased goods Zhang San Milk, bread, butter Li Si Milk, bread Wang Wu Bread, butter, eggs Zhao Liu Milk, eggs Qian Qi Bread, eggs

[0110] Assuming that frequent itemsets are found according to a minimum support threshold, such as 30%, this means that an itemset must appear in at least 2 transactions. We can get:

[0111] 1. Support for a single product:

[0112] Milk: 3 times (transactions 1, 2, 4)

[0113] Bread: 4 times (transactions 1, 2, 3, 5)

[0114] Butter: 2 times (transactions 1, 3)

[0115] Eggs: 3 times (transactions 3, 4, 5)

[0116] 2. Frequent itemsets (support ≥ 30%):

[0117] {Bread}: appears 4 times

[0118] {Milk}: appears 3 times

[0119] {Egg}: Appears 3 times

[0120] {bread, eggs}: appears 2 times

[0121] Next, association rules are generated from frequent item sets. For each frequent item set A, a rule A->B can be generated, where B is all possible combinations of subsets of A.

[0122] Generate a rule from {bread, milk}: If you buy bread, you are likely to buy milk.

[0123] Secondly, after generating association rules according to frequent item sets in the above steps, the accuracy of the association rules is evaluated using confidence and lift. First, the confidence is calculated according to the above association rules, and then the lift is calculated to obtain association rules with higher strength.

[0124] Optionally, in this embodiment, a corresponding project relationship pattern table is constructed according to data association rules.

[0125] Create a table to record the relationship between items. The table usually contains the following columns:

[0126] Project portfolio: The collection of projects involved in a rule.

[0127] Support: How often a combination of items appears in all transactions.

[0128] Confidence: The conditional probability of the occurrence of the subsequent term given the occurrence of the preceding term.

[0129] Lift: The strength of the rule, which is the ratio of the confidence to the frequency of occurrence of the consequent in all transactions.

[0130] For example, take the above supermarket dataset as an example:

[0131] Identify the transaction data set: {bread, butter, milk}

[0132] Find frequent itemsets based on support: {bread, milk}

[0133] Association rule generation: Generate rules that meet thresholds, such as "If you buy bread, you are likely to buy milk."

[0134] The data used in the above generation rules are used to construct the project relationship model table as follows:

[0135] Item combination: {bread, milk}

[0136] Support: 5%

[0137] Confidence level: 70%

[0138] Lift: 2.35

[0139] Optionally, in this embodiment, after the project relationship pattern table is constructed, the project relationship pattern table may also be displayed in a visual interface to facilitate understanding and analysis.

[0140] It is understandable that after the above steps, the project relationship pattern table can fully discover the association rules between enterprise data, provide valuable business insights for enterprises, and help optimize product layout and marketing strategies. For example, according to the pattern table, supermarkets can place milk near the bread area to increase milk sales.

[0141] Step S103: extract the associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and the preset abnormal threshold, and output the abnormal associated data features to the visualization interface for alarm display.

[0142] Optionally, in this embodiment, the purpose of this step is to identify outliers in the associated data through the isolation forest algorithm and to display the outliers as visual warnings to help enterprises identify possible risks.

[0143] Optionally, in this embodiment, the technical means of this step is to input the associated data features in the project pattern relationship table into a trained isolation forest model, so that the model can identify abnormal data therein.

[0144] For example, if a customer buys milk when buying bread, the confidence level is 70%. However, during the operation of a supermarket, the confidence level of the associated data in the item pattern relationship table obtained from the operation data is 5%. The item pattern and confidence level abnormality are displayed as a visual alarm. The supermarket owner finds that the problem is due to insufficient bread inventory, so he immediately replenishes the inventory to increase profits.

[0145] Optionally, in this embodiment, the steps for implementing the technical means are: first, extracting the associated data features in the project relationship model table, inputting the associated data features into the feature selection layer of a preset isolation forest algorithm model to perform feature selection operations, and determining corresponding selection features.

[0146] Specifically, this step introduces randomness in feature selection and randomly selects a feature for segmentation. This randomness reduces the reliance on specific features and helps the algorithm capture global abnormal patterns in the data. After selecting the feature, the algorithm randomly selects a split point between the maximum and minimum values ​​of the feature. The data is divided into two parts according to this split point, one part is the samples with feature values ​​less than the split point, and the other part is the samples with feature values ​​greater than or equal to the split point. This randomness further increases the generalization ability of the algorithm, enabling it to explore the data space from different angles.

[0147] It can be understood that by randomly selecting features and split points, the isolation forest algorithm can perform random partitioning in the feature space. This partitioning method helps the algorithm to quickly identify outliers because outliers are usually sparsely distributed in the feature space.

[0148] Secondly, the selected features are input into the decision tree construction layer of the preset isolation forest algorithm model, and a decision forest construction operation is performed according to a preset decision tree quantity parameter to determine a corresponding decision forest.

[0149] Specifically, this step is to input the selected features obtained after the above random segmentation into the decision tree construction layer as the root node of the decision tree, and repeat the above random selection and segmentation process to recursively construct the decision tree until the preset decision tree depth is reached. In the above process, each decision tree performs feature segmentation on different random feature points. Among them, the decision tree depth is a hyperparameter, which is determined according to the performance and evaluation in the model training process. The selection of hyperparameters plays a vital role in the model training process. The isolation forest algorithm forms a forest by constructing multiple isolated trees. The construction of each tree is independent and can be carried out in parallel, which improves the efficiency of the algorithm. Among them, the number of decision trees is a preset quantity parameter, which is a hyperparameter of the isolation forest algorithm model. The size of this parameter determines the training performance of the model. More trees will increase the computing time and resource consumption. For general data sets, 100 trees are selected for initialization training. More trees may increase the complexity of the model and need to be balanced to avoid overfitting.

[0150] Optionally, the above decision tree stopping condition can also be set as the number of samples in the node is less than a certain threshold, or all samples of the current node have the same value on the selected feature, which does not affect the implementation of the embodiment of the present invention.

[0151] Then, an abnormality evaluation operation is performed on each decision tree in the decision forest, and an integration operation is performed on the abnormality scores obtained through the abnormality evaluation operation to determine a corresponding integrated score.

[0152] Specifically, this step evaluates the anomaly of all isolated trees separately. For each tree, the length of the path (from the root node to the leaf node) of each data point in the tree can be used to calculate an anomaly score. Usually, this score is based on the "isolation path length" of the data point to the leaf node. Outliers tend to be isolated faster in the tree, so the path length is shorter.

[0153] The steps for calculating the anomaly score are as follows:

[0154] Calculate the path length: For each data point, in each isolation tree, calculate the number of edges required from the root node to isolate the data point (reach the leaf node), that is, the path length.

[0155] Calculate the average path length: For each data point, calculate the average of its path lengths in all isolated trees. This average reflects the abnormality of the data point. Usually, abnormal points have shorter average path lengths.

[0156] Anomaly score calculation: Using the relationship between the average path length and the average path length of the tree, an anomaly score is calculated for each data point. The anomaly score is a measure of the negative of the average path length.

[0157] Since each tree is built independently and each tree can assign an anomaly score to each data point in the dataset, one tree can score multiple data points. The final anomaly score for each data point is obtained by averaging or otherwise integrating the anomaly scores assigned by all trees to that data point.

[0158] Optionally, in this embodiment, the ensemble method uses weighted average ensemble to obtain the ensemble anomaly score for each data point. For trees with higher performance or accuracy, the weighted average can be calculated by assigning greater weights to the anomaly scores of these trees. The weights can be based on specific performance indicators of the tree, such as accuracy on the validation set.

[0159] Finally, the abnormal correlation data features output by the preset isolation forest algorithm model are determined according to the integrated score and the preset abnormal threshold, and the abnormal correlation data features are output to the visualization interface for alarm display.

[0160] Specifically, this step distinguishes normal points from abnormal points by selecting a threshold, where the threshold is determined based on the anomaly detection performance on the validation set. By comparing the comprehensive integrated score with the abnormal threshold, if the integrated abnormal score exceeds the threshold, it is classified as abnormal; if it is lower than the threshold, it is classified as normal. The abnormal value and its corresponding features are output to the visualization interface for alarm display.

[0161] For example, take the above supermarket sales data as an example:

[0162] It is known that there is a supermarket business project relationship model table as follows:

[0163] Item combination: {bread, milk}

[0164] Support: 5%

[0165] Confidence level: 5%

[0166] Lift: 2.35

[0167] Extract the features in the above relationship table and use the isolation forest model to calculate the anomaly score of each item pattern in the operational data set. When the bread + milk combination is detected with a confidence level of 5%, the model calculates an anomaly score of, for example, 0.8, assuming that the anomaly score threshold is 0.7.

[0168] On the visualization interface, the combination is highlighted in red, and an alert window pops up: "An abnormal confidence level has been detected. Please check the item combination: bread + milk."

[0169] This example demonstrates how this embodiment unifies data types and constructs data correlation, performs prediction and warning based on the correlation, guides enterprises to make decisions, and improves enterprise efficiency.

[0170] From the above description, it can be seen that the associated data identification and alarm method based on data visualization provided in the embodiment of the present application can determine the corresponding dimensional data and metric data by receiving data input by the user, determine the corresponding basic characteristics of the data according to the dimensional data, metric data and statistical analysis rules, input the data features into the associated learning algorithm, determine the corresponding project relationship pattern table, extract the associated data features in the project relationship pattern table, input the associated data features into the preset isolation forest algorithm model to determine the corresponding decision forest, perform abnormal evaluation operations on the decision trees in the decision forest respectively, perform integration operations on the abnormal scores obtained through the abnormal evaluation operations, determine the abnormal associated data features according to the integrated scores and the preset abnormal thresholds, and output the abnormal associated data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data correlation identification, thereby improving the timeliness of data analysis.

[0171] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 2 , and can also include the following:

[0172] Step S201: receiving data input by a user, determining whether the data input by the user conforms to a preset data type detection rule, and if so, determining a corresponding first data type according to the preset data type detection rule;

[0173] Step S202: If not, determining the corresponding second data type according to the data input by the user and the type inference function.

[0174] Step S203: Determine a corresponding data type according to the first data type and the second data type.

[0175] Optionally, in this embodiment, first, it is determined whether the data belongs to the preset data type detection rules based on the data characteristics input by the user. For example, if the existing standard data type detection rules: the field name "order number" is the "integer" type, then if the field name entered by the user is "order number", the data type is determined according to the rules.

[0176] Secondly, if the data characteristics of the user input do not conform to the rules, the type inference function is used to infer the type. For example, if the field name entered by the user is "order label" and does not conform to the existing standard data type detection rules, then it is inferred as "integer" type according to the type inference function.

[0177] Optionally, in the present embodiment, in the above data detection algorithm, the inference function is to use a special type inference function, which attempts to convert the data into different types and infers the most likely type based on success or failure. For example, if the field name entered by the user is "order label", which does not conform to the existing standard data type detection rules, the inference function uses a set of predefined rules to identify the type of the data field. For example, if the field value contains only numbers, it may be inferred to be an integer or floating point type.

[0178] Optionally, in this embodiment, in the above-mentioned data detection algorithm, the inference function can be replaced by a machine learning algorithm based on demand, and the machine learning algorithm is applied to learn the type of data fields, which involves training models to recognize different types of data patterns, and then using these models to infer the type of new data.

[0179] Optionally, in this embodiment, on the one hand, the best way to infer the data type is rule-based type inference, because most of the data input by the user is structured tabular data, and the rule-based method is effective enough, and does not require model training, saving time. On the other hand, if the data input by the user is disorganized, a machine learning algorithm is suitable for this case. Using a machine model to learn the data model in the user data saves time compared to manually defining rules. In general, based on the above method, before performing data detection, features that are helpful for type inference, such as numerical range, format, length, etc., are first extracted from the data, and the extracted features are compared with known data type patterns. Based on the matching results and possible rules or models, the most likely type of the data field is determined.

[0180] It can be understood that the above steps are to determine the data type. Correct data type determination can ensure the accuracy of the data and lay a solid foundation for subsequent data feature identification.

[0181] Through step S203, this embodiment successfully makes an accurate judgment on the data type, laying a solid foundation for further identifying data features according to the type.

[0182] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 3 , and can also include the following:

[0183] Step S301: extracting the field name in the data type, performing a field mapping operation according to the field name and a preset business field list, and determining corresponding dimension features and metric features;

[0184] Step S302: Determine corresponding dimension data and metric data according to the dimension feature, the metric feature and a preset business rule engine.

[0185] Optionally, in this embodiment, the determination of the dimension data and the metric data is achieved by the following steps:

[0186] 1. Dimensional feature identification:

[0187] Dimensional features are usually used in the context of classification, grouping or analysis, such as time, location, product category, etc.

[0188] Automatically identify dimension features by matching keywords in field names with business rules.

[0189] For example, a field containing "ID", "Name", or "Email" might be considered a dimension.

[0190] 2. Metric feature recognition:

[0191] Metric characteristics are quantifiable values ​​used to measure or evaluate, such as sales, profit, volume, etc.

[0192] Automatically identify metric features by matching keywords in field names with business rules.

[0193] For example, a field containing terms such as "sales", "quantity", etc. might be identified as a measure.

[0194] 3. Business rule engine application:

[0195] Use the preset business rules engine to further refine field classification and ensure data accuracy and applicability. The engine can apply the above rules to automatically classify data fields as dimensional data or metric data.

[0196] 4. Determination of dimensional data and measurement data:

[0197] Based on the field mapping results and the output of the business rule engine, it is ultimately determined which fields belong to dimensional data and which belong to metric data.

[0198] It should be noted that for measurement data, its dimension needs to be determined (such as meter, kilogram, RMB, etc.). Machine learning models are usually used to identify and convert measurement data of different dimensions to a unified scale.

[0199] For example, let's say a large retail chain wants to analyze its sales data:

[0200] Extract fields: product number, product name, category, sales date, sales quantity, unit price

[0201] Feature Identification: "Category" and "Sales Date" are identified as dimension features, and "Sales Quantity" and "Unit Price" are identified as measure features.

[0202] Apply the business rule engine: Determine that the "Sales Date" field needs to be further subdivided into dimensions by year, quarter, and month.

[0203] Data determination: According to mapping and business rules, "category" and "sales date" (broken down into year, quarter, and month) are classified as dimension data; "sales quantity" and "unit price" are classified as measure data.

[0204] It can be understood that the above steps are a process of identifying and classifying data. The data defined by the user according to different rules can be re-identified and classified according to its data characteristics. The dimensional data and metric data obtained in this way have unified data characteristics, laying the foundation for the subsequent discovery of data correlation based on data characteristics.

[0205] Through step S302, this embodiment realizes the unified definition of data, laying a foundation for the subsequent discovery of the association between data and the construction of the project relationship model table.

[0206] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 4 , and can also include the following:

[0207] Step S401: performing statistical analysis operations and visual analysis operations on the dimension data and the metric data to respectively determine corresponding data center trends and data distribution trends;

[0208] Step S402: Determine corresponding basic data characteristics according to the data center trend and the data distribution trend.

[0209] Optionally, in this embodiment, first, descriptive statistical analysis is performed on the measurement data, including calculating the mean, median, standard deviation, minimum and maximum values, etc., to understand the central tendency and dispersion of the data. For dimensional data, the trend under different dimensions can be identified by grouping the mean and sum of the measurement data. Through statistical analysis, the central tendency of the data can be obtained. Among them, the statistical analysis uses descriptive statistical analysis, which can be achieved by a preset statistical analysis algorithm.

[0210] Secondly, use bar charts, line charts, scatter plots and other charts to show the distribution and trend of metric data. Use box plots to show the distribution of data, including median, quartiles and outliers. Use heat maps or tree diagrams to show the relationship between data of different dimensions. Through visual analysis, the distribution trend of data can be obtained. Visual analysis can be performed through libraries such as Matplotlib, Seaborn or Plotly for data visualization.

[0211] Optionally, in this embodiment, an example is given:

[0212] Dimensional data: time (year, month), region (city), product type

[0213] Metrics: Sales, Sales Quantity

[0214] Statistical analysis: Calculate the mean, median, standard deviation, minimum, and maximum of sales and sales quantity. Determine the central tendency and dispersion of the data. Draw a line graph for time series data (such as monthly or annual sales data) to observe the trend of sales and sales quantity over time. Analyze sales and sales quantity by region and product type, using grouping statistics.

[0215] Visual analysis: Use bar charts to show sales and sales quantities by region or product type. Use box plots to show the distribution of sales and sales quantities and identify outliers. Use scatter plots to analyze the relationship between sales and sales quantities and check whether there is a linear relationship.

[0216] It is understandable that through the above analysis, the trends between the various data are obtained, and the basic characteristics between the products can be obtained by analyzing the trends, for example, determining which products have the characteristics of high sales and frequent purchases. These characteristics lay a solid foundation for the subsequent extraction of characteristic features and the discovery of the correlation relationship therein.

[0217] Optionally, in this embodiment, based on the results of trend analysis, basic characteristics of the data are determined, such as stability, periodicity, outliers, etc. Useful data features are extracted from the basic characteristics of the data, which are meaningful for association rule learning, such as: the frequency of item occurrence, the co-occurrence frequency between items, the volatility or stability index of the data.

[0218] It can be understood that through the above steps, a data feature containing data trends is obtained, which lays the foundation for the subsequent discovery of the correlation between data through association algorithms.

[0219] Through step S402, this embodiment successfully obtains the trends and characteristics between the data, laying a foundation for subsequently discovering data associations from the characteristics.

[0220] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 5 , and can also include the following:

[0221] Step S501: inputting the associated data features into a feature selection layer of a preset isolation forest algorithm model to perform a random feature selection operation to determine corresponding random features;

[0222] Step S502: performing a random segmentation point selection operation on the random feature to determine a corresponding random segmentation point;

[0223] Step S503: determining corresponding selection features according to the random features and the random segmentation points.

[0224] Optionally, in this embodiment, this step is to introduce randomness in feature selection and randomly select a feature for segmentation. This randomness reduces the dependence on specific features and helps the algorithm capture global abnormal patterns in the data. After selecting the feature, the algorithm randomly selects a split point between the maximum and minimum values ​​of the random feature, and the data is divided into two parts according to this split point, one part is the sample with a feature value less than the split point, and the other part is the sample with a feature value greater than or equal to the split point. This randomness further increases the generalization ability of the algorithm, enabling it to explore the data space from different angles.

[0225] It can be understood that by randomly selecting features and split points, the isolation forest algorithm can perform random partitioning in the feature space. This partitioning method helps the algorithm to quickly identify outliers because outliers are usually sparsely distributed in the feature space.

[0226] Through step S503, this embodiment successfully determines the features through random selection and random segmentation, which lays a foundation for subsequent detection of outliers.

[0227] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 6 , and can also include the following:

[0228] Step S601: inputting the selected features into the decision tree construction layer of the preset isolation forest algorithm model, performing a decision tree construction operation according to a preset recursive rule and a preset depth parameter, and determining a corresponding decision tree;

[0229] Step S602: determining a corresponding decision forest according to the decision tree and a preset decision tree quantity parameter.

[0230] Specifically, this step is to input the selected features obtained after the above random segmentation into the decision tree construction layer as the root node of the decision tree, and repeat the above random selection and segmentation process to recursively construct the decision tree until the preset decision tree depth is reached. In the above process, each decision tree performs feature segmentation on different random feature points. Among them, the decision tree depth is a hyperparameter, which is determined according to the performance and evaluation in the model training process. The selection of hyperparameters plays a vital role in the model training process. The isolation forest algorithm forms a forest by constructing multiple isolated trees. The construction of each tree is independent and can be carried out in parallel, which improves the efficiency of the algorithm. Among them, the number of decision trees is a preset quantity parameter, which is a hyperparameter of the isolation forest algorithm model. The size of this parameter determines the training performance of the model. More trees will increase the computing time and resource consumption. For general data sets, 100 trees are selected for initialization training. More trees may increase the complexity of the model and need to be balanced to avoid overfitting.

[0231] Optionally, the above decision tree stopping condition can also be set as the number of samples in the node is less than a certain threshold, or all samples of the current node have the same value on the selected feature, which does not affect the implementation of the embodiment of the present invention.

[0232] Through step S602, this embodiment successfully constructs a decision forest for the input features by random crossover, laying a foundation for subsequent outlier evaluation for each feature point.

[0233] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 7 , and can also include the following:

[0234] Step S701: Calculate the path lengths of data points of the decision trees in the decision forest respectively to determine the corresponding first path lengths;

[0235] Step S702: Calculate the average path lengths of the decision trees in the decision forest respectively to determine the corresponding second path lengths;

[0236] Step S703: determining a corresponding abnormality score according to the first path length and the second path length.

[0237] Optionally, in this embodiment, this step is to evaluate the anomaly of all isolated trees separately. For each tree, the length of the path (the path from the root node to the leaf node) of each data point in the tree can be used to calculate an anomaly score. Usually, this score is based on the "isolation path length" of the data point to the leaf node. Outliers tend to be isolated faster in the tree, so the path length is shorter.

[0238] The steps for calculating the anomaly score are as follows:

[0239] Calculate the path length: For each data point, in each isolation tree, calculate the number of edges required from the root node to isolate the data point (reach the leaf node), that is, the path length.

[0240] Calculate the average path length: For each data point, calculate the average of its path lengths in all isolated trees. This average reflects the abnormality of the data point. Usually, abnormal points have shorter average path lengths.

[0241] Anomaly score calculation: Using the relationship between the average path length and the average path length of the tree, an anomaly score is calculated for each data point. The anomaly score is a measure of the negative of the average path length.

[0242] Since each tree is built independently and each tree can assign an anomaly score to each data point in the dataset, one tree can score multiple data points. The final anomaly score for each data point is obtained by averaging or otherwise integrating the anomaly scores assigned by all trees to that data point.

[0243] Through step S703, this embodiment successfully obtains the evaluation score of each decision tree for the feature point, laying a foundation for the subsequent integration of the evaluation scores and finding outliers.

[0244] In one embodiment of the method for identifying and warning related data based on data visualization of the present application, see Figure 8 , and can also include the following:

[0245] Step S801: performing a weighted average integration operation on the anomaly scores obtained through the anomaly evaluation operation to obtain a corresponding integration score;

[0246] Step S802: determining whether the integration score exceeds a preset abnormal threshold, and if so, outputting the associated data feature corresponding to the integration score as an abnormal associated data feature.

[0247] Optionally, in this embodiment, the ensemble method uses weighted average ensemble to obtain an ensemble anomaly score for each data point. For trees with higher performance or accuracy, a weighted average can be calculated by assigning greater weights to the anomaly scores of these trees. The weights can be based on specific performance indicators of the tree, such as accuracy on a validation set.

[0248] Finally, the abnormal correlation data features output by the preset isolation forest algorithm model are determined according to the integrated score and the preset abnormal threshold, and the abnormal correlation data features are output to the visualization interface for alarm display.

[0249] Specifically, this step distinguishes normal points from abnormal points by selecting a threshold, where the threshold is determined based on the anomaly detection performance on the validation set. By comparing the comprehensive integrated score with the abnormal threshold, if the integrated abnormal score exceeds the threshold, it is classified as abnormal; if it is lower than the threshold, it is classified as normal. The abnormal value and its corresponding features are output to the visualization interface for alarm display.

[0250] For example, take the above supermarket sales data as an example:

[0251] It is known that there is a supermarket business project relationship model table as follows:

[0252] Item combination: {bread, milk}

[0253] Support: 5%

[0254] Confidence level: 5%

[0255] Lift: 2.35

[0256] Extract the features in the above relationship table and use the isolation forest model to calculate the anomaly score of each item pattern in the operational data set. When the bread + milk combination is detected with a confidence level of 5%, the model calculates an anomaly score of, for example, 0.8, assuming that the anomaly score threshold is 0.7.

[0257] On the visualization interface, the combination is highlighted in red, and an alert window pops up: "An abnormal confidence level has been detected. Please check the item combination: bread + milk."

[0258] Through step S802, this embodiment successfully obtains abnormal values ​​of associated data and displays alarms, thereby helping enterprises make decisions and improving the efficiency of enterprise data utilization.

[0259] In order to perform real-time prediction and warning based on data association identification and improve the timeliness of data analysis, the present application provides an embodiment of a device for identifying and warning related data based on data visualization, which is used to implement all or part of the contents of the method for identifying and warning related data based on data visualization. Fig. 9 The associated data recognition and warning device based on data visualization specifically includes the following contents:

[0260] The data identification module 10 is used to receive data input by a user, perform a data detection operation on the data input by the user to determine a corresponding data type, perform a data identification operation according to the data type to determine corresponding dimension data and metric data;

[0261] An association rule determination module 20 is used to determine the corresponding basic characteristics of the data according to the dimension data, the measurement data and the preset statistical analysis rules, input the data features in the basic characteristics of the data into the association learning algorithm, determine the corresponding data association rules, and construct the corresponding project relationship pattern table according to the data association rules;

[0262] The associated abnormality alarm module 30 is used to extract the associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform an abnormality evaluation operation on the decision trees in the decision forest respectively, perform an integration operation on the abnormality scores obtained through the abnormality evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and the preset abnormal threshold, and output the abnormal associated data features to the visualization interface for alarm display.

[0263] From the above description, it can be seen that the associated data identification and alarm device based on data visualization provided in the embodiment of the present application can determine the corresponding dimensional data and metric data by receiving data input by the user, determine the corresponding basic characteristics of the data according to the dimensional data, metric data and statistical analysis rules, input the data features into the associated learning algorithm, determine the corresponding project relationship pattern table, extract the associated data features in the project relationship pattern table, input the associated data features into the preset isolation forest algorithm model to determine the corresponding decision forest, perform abnormal evaluation operations on the decision trees in the decision forest respectively, perform integration operations on the abnormal scores obtained through the abnormal evaluation operations, determine the abnormal associated data features according to the integrated scores and the preset abnormal thresholds, and output the abnormal associated data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data correlation identification, thereby improving the timeliness of data analysis.

[0264] From the hardware level, in order to perform real-time prediction and alarm based on data association identification and improve the timeliness of data analysis, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the associated data identification and alarm method based on data visualization, and the electronic device specifically includes the following contents:

[0265] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the method for identifying and warning associated data based on data visualization and related devices such as core business systems, user terminals and related databases; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the method for identifying and warning associated data based on data visualization and the embodiment of the method for identifying and warning associated data based on data visualization in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0266] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0267] In practical applications, part of the method for identifying and warning associated data based on data visualization can be executed on the electronic device side as described above, or all operations can be completed in the client device. The selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0268] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0269] Fig.10 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.10 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.10 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0270] In one embodiment, the function of the associated data identification alarm method based on data visualization can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0271] Step S101: receiving data input by a user, performing a data detection operation on the data input by the user to determine a corresponding data type, performing a data recognition operation according to the data type to determine corresponding dimension data and metric data;

[0272] Step S102: determining corresponding basic data characteristics according to the dimension data, the measurement data and preset statistical analysis rules, inputting data features in the basic data characteristics into an association learning algorithm, determining corresponding data association rules, and constructing a corresponding project relationship pattern table according to the data association rules;

[0273] Step S103: extract the associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and the preset abnormal threshold, and output the abnormal associated data features to the visualization interface for alarm display.

[0274] From the above description, it can be seen that the electronic device provided in the embodiment of the present application receives data input by the user, determines the corresponding dimensional data and metric data, determines the corresponding basic characteristics of the data according to the dimensional data, metric data and statistical analysis rules, inputs the data features into the association learning algorithm, determines the corresponding project relationship pattern table, extracts the associated data features in the project relationship pattern table, inputs the associated data features into the preset isolation forest algorithm model to determine the corresponding decision forest, performs abnormal evaluation operations on the decision trees in the decision forest respectively, performs integration operations on the abnormal scores obtained through the abnormal evaluation operations, determines the abnormal associated data features according to the integrated scores and the preset abnormal thresholds, and outputs the abnormal associated data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data correlation identification, thereby improving the timeliness of data analysis.

[0275] In another embodiment, the associated data identification and alarm method based on data visualization can be configured separately from the central processing unit 9100. For example, the associated data identification and alarm method based on data visualization can be configured as a chip connected to the central processing unit 9100, and the function of the associated data identification and alarm method based on data visualization can be realized through the control of the central processing unit.

[0276] like Fig.10 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.10 In addition, the electronic device 9600 may also include Fig.10 For components not shown, reference may be made to the prior art.

[0277] like Fig.10 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0278] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0279] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0280] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0281] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0282] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0283] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module, etc. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0284] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method for identifying and warning associated data based on data visualization in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the method for identifying and warning associated data based on data visualization in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0285] Step S101: receiving data input by a user, performing a data detection operation on the data input by the user to determine a corresponding data type, performing a data recognition operation according to the data type to determine corresponding dimension data and metric data;

[0286] Step S102: determining corresponding basic data characteristics according to the dimension data, the measurement data and preset statistical analysis rules, inputting data features in the basic data characteristics into an association learning algorithm, determining corresponding data association rules, and constructing a corresponding project relationship pattern table message analysis model according to the data association rules;

[0287] Step S103: extract the associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and the preset abnormal threshold, and output the abnormal associated data features to the visualization interface for alarm display.

[0288] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application receives data input by the user, determines the corresponding dimensional data and metric data, determines the corresponding basic characteristics of the data according to the dimensional data, metric data and statistical analysis rules, inputs the data features into the association learning algorithm, determines the corresponding project relationship pattern table, extracts the associated data features in the project relationship pattern table, inputs the associated data features into the preset isolation forest algorithm model to determine the corresponding decision forest, performs abnormal evaluation operations on the decision trees in the decision forest respectively, performs integration operations on the abnormal scores obtained through the abnormal evaluation operations, determines the abnormal associated data features according to the integrated scores and the preset abnormal thresholds, and outputs the abnormal associated data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data correlation identification, thereby improving the timeliness of data analysis.

[0289] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the method for identifying and warning associated data based on data visualization in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the method for identifying and warning associated data based on data visualization are implemented. For example, the computer program / instruction implements the following steps:

[0290] Step S101: receiving data input by a user, performing a data detection operation on the data input by the user to determine a corresponding data type, performing a data recognition operation according to the data type to determine corresponding dimension data and metric data;

[0291] Step S102: determining corresponding basic data characteristics according to the dimension data, the measurement data and preset statistical analysis rules, inputting data features in the basic data characteristics into an association learning algorithm, determining corresponding data association rules, and constructing a corresponding project relationship pattern table according to the data association rules;

[0292] Step S103: extract the associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and the preset abnormal threshold, and output the abnormal associated data features to the visualization interface for alarm display.

[0293] From the above description, it can be seen that the computer program product provided by the embodiment of the present application receives data input by the user, determines the corresponding dimensional data and metric data, determines the corresponding basic characteristics of the data according to the dimensional data, metric data and statistical analysis rules, inputs the data features into the association learning algorithm, determines the corresponding project relationship pattern table, extracts the associated data features in the project relationship pattern table, inputs the associated data features into the preset isolation forest algorithm model to determine the corresponding decision forest, performs abnormal evaluation operations on the decision trees in the decision forest respectively, performs integration operations on the abnormal scores obtained through the abnormal evaluation operations, determines the abnormal associated data features according to the integrated scores and the preset abnormal thresholds, and outputs the abnormal associated data features to the visualization interface for alarm display, thereby enabling real-time prediction and alarm based on data correlation identification, thereby improving the timeliness of data analysis.

[0294] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0295] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0296] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0297] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0298] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for identifying and warning related data based on data visualization, characterized in that: The method comprises: Receive data input by a user, perform a data detection operation on the data input by the user to determine a corresponding data type, perform a data recognition operation according to the data type to determine corresponding dimension data and metric data; Determine the corresponding basic data characteristics according to the dimension data, the measurement data and the preset statistical analysis rules, input the data features in the basic data characteristics into the association learning algorithm, determine the corresponding data association rules, and construct the corresponding project relationship pattern table according to the data association rules; Extract the associated data features in the project relationship pattern table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and a preset anomaly threshold, and output the abnormal associated data features to a visualization interface for alarm display.

2. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The receiving of data input by a user, performing a data detection operation on the data input by the user, and determining a corresponding data type includes: Receiving data input by a user, determining whether the data input by the user conforms to a preset data type detection rule, and if so, determining a corresponding first data type according to the preset data type detection rule; If not, determining the corresponding second data type according to the data input by the user and the type inference function; According to the first data type and the second data type, a corresponding data type is determined.

3. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The performing of data identification operations according to the data types to determine corresponding dimension data and metric data includes: Extracting the field name in the data type, performing field mapping operation according to the field name and a preset business field list, and determining corresponding dimension features and metric features; Corresponding dimensional data and metric data are determined according to the dimensional features, the metric features and a preset business rule engine.

4. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The determining of the corresponding basic characteristics of the data according to the dimensional data, the metric data and the preset statistical analysis rules includes: Performing statistical analysis operations and visual analysis operations on the dimension data and the metric data to respectively determine corresponding data center trends and data distribution trends; Corresponding basic data characteristics are determined according to the data center trend and the data distribution trend.

5. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The inputting the associated data features into the feature selection layer of the preset isolation forest algorithm model to perform feature selection operation and determine the corresponding selection features, including: Inputting the associated data features into a feature selection layer of a preset isolation forest algorithm model to perform a random feature selection operation to determine corresponding random features; Performing a random segmentation point selection operation on the random feature to determine a corresponding random segmentation point; A corresponding selection feature is determined according to the random feature and the random segmentation point.

6. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The step of inputting the selected features into the decision tree construction layer of the preset isolation forest algorithm model, performing a decision forest construction operation according to a preset decision tree quantity parameter, and determining a corresponding decision forest comprises: Input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision tree construction operation according to a preset recursive rule and a preset depth parameter, and determine a corresponding decision tree; A corresponding decision forest is determined according to the decision tree and a preset decision tree quantity parameter.

7. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The performing abnormal evaluation operations on the decision trees in the decision forest respectively includes: Calculating the path lengths of data points of the decision trees in the decision forest respectively to determine the corresponding first path lengths; Calculating the average path lengths of the decision trees in the decision forest respectively to determine the corresponding second path lengths; A corresponding anomaly score is determined according to the first path length and the second path length.

8. The method for identifying and warning related data based on data visualization according to claim 1 is characterized in that: The integrating operation is performed on the anomaly scores obtained through the anomaly evaluation operation to determine the corresponding integrated scores, and the abnormal correlation data features output by the preset isolation forest algorithm model are determined according to the integrated scores and the preset anomaly threshold, including: Performing a weighted average integration operation on the anomaly scores obtained through the anomaly evaluation operation to obtain a corresponding integration score; It is determined whether the integration score exceeds a preset abnormal threshold, and if so, the associated data feature corresponding to the integration score is output as an abnormal associated data feature.

9. A device for identifying and warning related data based on data visualization, characterized in that: The device comprises: A data identification module is used to receive data input by a user, perform a data detection operation on the data input by the user, determine a corresponding data type, perform a data identification operation according to the data type, and determine corresponding dimension data and metric data; An association rule determination module is used to determine the corresponding basic characteristics of the data according to the dimension data, the measurement data and the preset statistical analysis rules, input the data features in the basic characteristics of the data into the association learning algorithm, determine the corresponding data association rules, and construct the corresponding project relationship pattern table according to the data association rules; An associated anomaly alarm module is used to extract associated data features in the project relationship model table, input the associated data features into the feature selection layer of the preset isolation forest algorithm model for feature selection operation, determine the corresponding selected features, input the selected features into the decision tree construction layer of the preset isolation forest algorithm model, perform a decision forest construction operation according to a preset decision tree quantity parameter, determine the corresponding decision forest, perform anomaly evaluation operations on the decision trees in the decision forest respectively, perform an integration operation on the anomaly scores obtained through the anomaly evaluation operation, determine the corresponding integration score, determine the abnormal associated data features output by the preset isolation forest algorithm model according to the integration score and a preset anomaly threshold, and output the abnormal associated data features to a visualization interface for alarm display.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for identifying and warning associated data based on data visualization described in any one of claims 1 to 7 are implemented.