Data analysis method based on association rule and regression model

Abstract

The invention relates to a data analysis method based on association rules and a regression model, which is carried out according to the following sequence: obtaining a data set to be analyzed from a data source, carrying out preprocessing operation on the data set, discovering correlation among data, extracting features in the data set, and establishing a relation between the features and target variables by utilizing the regression model. The method comprises the steps of establishing a regression model, carrying out model training and evaluation, carrying out optimization operation on the established regression model to improve the prediction performance of the model, carrying out prediction and analysis on new data by utilizing the optimized regression model, and providing decision support and business guidance. The analysis problem of large-scale data can be effectively solved, and the efficiency and accuracy of data analysis are improved.

Description

Technical Field

[0001] The present invention relates to a data analysis method based on association rules and regression models, which is a data analysis method that can effectively analyze large-scale data and can be used in data analysis, business decision-making, etc. of enterprises, governments and other organizations. Background Art

[0002] With the development of information technology and the popularization of the Internet, data has become an important resource for management and decision-making of organizations such as enterprises and governments. The emergence of big data has made data analysis and mining more important. In the decision-making process of organizations such as enterprises and governments, data analysis technology has become an indispensable means. Traditional data analysis methods mainly rely on manual analysis, statistical methods and other means. This method has problems such as sample error and high time cost, and it is difficult to meet the analysis needs of large-scale data. Therefore, a data analysis method that can effectively process large-scale data is needed. Association rule mining and regression models are commonly used methods in the field of data analysis, which can effectively mine the association relationship between data and establish data models for prediction and analysis. Therefore, the present invention proposes a data analysis method based on association rules and regression models to solve the problems existing in large-scale data analysis. Summary of the invention

[0003] The present invention proposes a data analysis method based on association rules and regression models, which mainly includes the following steps: Data acquisition: Obtain the data set to be analyzed from the data source.

[0004] Data preprocessing: Perform preprocessing operations such as cleaning, deduplication, missing value processing, and outlier processing on the data set to ensure data quality.

[0005] Association rule mining: Use association rule mining algorithms such as the Apriori algorithm to discover the correlation between data by mining frequent item sets and association rules.

[0006] Feature engineering: Extract features from the data set based on the mined association rules and domain knowledge, including numerical features and categorical features.

[0007] Build a regression model: Use regression models such as linear regression and logistic regression to link features with target variables, and perform model training and evaluation.

[0008] Model optimization: Perform optimization operations such as parameter adjustment and feature selection on the established regression model to improve the prediction performance of the model.

[0009] Model application: Use the optimized regression model to predict and analyze new data to provide decision support and business guidance.

[0010] The data analysis method proposed in the present invention fully utilizes the advantages of association rule mining and regression models, can effectively handle analysis problems of large-scale data, and improve the efficiency and accuracy of data analysis.

[0011] To achieve the above purpose, the technical solution adopted by the present invention is as follows: 1. Data Acquisition Data acquisition is the first step in data analysis, which is to obtain the data set to be analyzed from the data source. The data source can include data in the internal systems of enterprises, governments and other organizations, or it can use public data sources such as open data to obtain data. Data acquisition methods include manual data capture, API interface data acquisition, database query and other methods. When acquiring data, attention should be paid to the quality and confidentiality of the data to ensure the security and reliability of the data.

[0012] 2. Data Preprocessing Data preprocessing is an important step in data analysis. For problems such as missing values ​​and outliers in the data set, preprocessing operations are required to ensure data quality. The steps of data preprocessing include data cleaning, deduplication, missing value filling, outlier processing, etc. Among them, data cleaning includes operations such as removing duplicate data and processing outlier data; data deduplication refers to deduplication of duplicate data in the data set; missing value filling refers to filling the missing values ​​in the data set; outlier processing refers to processing the outliers in the data set, such as deleting outliers.

[0013] 3. Association Rule Mining Association rule mining is based on item sets and association rules in a data set, and discovers the correlation between data by mining frequent item sets and association rules. In association rule mining, commonly used algorithms include Apriori algorithm and FP-growth algorithm. Apriori algorithm is an association rule mining algorithm based on candidate sets, which discovers frequent item sets and association rules by scanning the data set multiple times; FP-growth algorithm is an association rule mining algorithm based on tree structure, which can effectively discover frequent item sets and association rules.

[0014] 4. Feature Engineering Feature engineering is the process of converting raw data into features that can be used for modeling. In feature engineering, it is necessary to extract features from the data set based on the association rules and domain knowledge that have been mined. Features in the data set can include numerical features and categorical features. Numerical features can be used to calculate statistics such as mean and standard deviation through statistical methods, and data normalization and standardization can also be performed; categorical features can be processed by one-hot encoding, label encoding, etc. The purpose of feature engineering is to improve the prediction accuracy and generalization ability of the model.

[0015] 5. Building a regression model Building a regression model is the process of establishing a connection between features and target variables, and training and evaluating the model. Regression models mainly include linear regression, logistic regression and other models. When building a regression model, factors such as the interpretability and predictive performance of the model need to be considered. When training the model, the data set needs to be divided into a training set and a test set. The training set is used to train the model, and the test set is used to evaluate the predictive performance of the model. Model evaluation indicators include mean absolute error (MAE), mean square error (MSE), coefficient of determination (R2), etc.

[0016] 6. Model Optimization Model optimization is to perform optimization operations such as parameter adjustment and feature selection on the established regression model to improve the prediction performance of the model. Commonly used model optimization methods include grid search and cross validation. Grid search is a parameter search method based on exhaustive enumeration, which searches for the optimal parameter combination by traversing the parameter combinations of the model; cross validation is a method to evaluate the generalization ability of the model, which divides the data set into a training set and a validation set, performs model training and validation multiple times, and finally obtains the average performance index of the model.

[0017] 7. Model Application Model application is to apply the established regression model to actual business for prediction and decision-making. When applying the model, new data sets need to be input into the model for prediction, prediction results are obtained, and decisions are made based on the prediction results. At the same time, the model needs to be monitored and maintained, and the model needs to be updated or reconstructed in a timely manner to ensure the prediction performance and stability of the model.

[0018] In order to better explain the analysis process of the present invention, two technical solutions will be described in detail: association rule mining and regression analysis.

[0019] Association rule mining Data preprocessing Before association rule mining, the data was preprocessed. The Python programming language and Pandas and NumPy libraries were used for data preprocessing. The original dataset was first loaded into a Pandas DataFrame, and missing values ​​and duplicate rows were removed. Then, some basic statistical analysis was performed on the data, including calculating the mean, median, maximum, and minimum values ​​of each variable. Finally, the numerical variables were standardized to eliminate scale differences and ensure the normal distribution of the data.

[0020] Association rule mining Next, we use the Apriori algorithm to mine association rules for the financial data set. The Apriori algorithm is a mining method based on frequent itemsets, which can find frequent itemsets in a data set and generate association rules from them. In this data case, the Python programming language and the mlxtend library are used to implement the Apriori algorithm. Convert the data set into a form suitable for the Apriori algorithm. Convert each variable into a binary encoding, where each binary bit indicates whether the variable has a certain value. For example, if a variable has three values ​​(A, B, C), convert it into three binary variables (A, B, C), where each variable indicates whether the variable has the corresponding value. Then use the Apriori algorithm to find frequent itemsets. Set the minimum support to 0.1, and only itemsets that appear in at least 10% of the transactions will be considered frequent itemsets. Through the Apriori algorithm, a set of frequent itemsets is obtained, which includes variables such as sales, cost, and profit. Use the association rule algorithm to generate association rules from frequent itemsets. Set the minimum confidence to 0.5, which means that only association rules with a confidence greater than 50% will be retained. Through the association rule algorithm, a set of association rules is obtained, which includes the association between sales and profit and the association between sales, cost and profit.

[0021] Results Analysis Through association rule mining, the relationship between sales and profits, the relationship between sales and profits, and the relationship between costs and profits are discovered. The conclusions of the analysis can help business managers better understand financial data and develop more effective management strategies.

[0022] Regression analysis Data preprocessing Before performing regression analysis, the data was preprocessed. The Python programming language and Pandas and NumPy libraries were used to perform data preprocessing. The original dataset was first loaded into a Pandas DataFrame, and missing values ​​and duplicate rows were removed. Then some basic statistical analysis was performed on the data, including calculating the mean, median, maximum, and minimum value of each variable. Finally, the numerical variables were standardized to eliminate scale differences and ensure the normal distribution of the data.

[0023] Regression model building Use linear regression model to perform regression analysis on financial data. Linear regression model is a mathematical model used to establish linear relationship, which can be used to predict and explain the relationship between variables. Use Python programming language and Scikit-learn library to build linear regression model.

[0024] The data set is divided into a training set and a test set, and the training set is used to fit the linear regression model. The least squares method is used to fit the model. The least squares method is a common regression analysis method that fits the model by minimizing the residual sum of squares. The test set is then used to evaluate the performance of the model. The root mean square error (RMSE) and R-squared value are used to evaluate the predictive ability of the model. RMSE refers to the square root of the mean of the difference between the actual value and the predicted value, which can be used to measure the prediction error of the model. The R-squared value refers to the ability of the fitted line to explain the overall variance, which can be used to measure the fitting effect of the model.

[0025] Results Analysis Through regression analysis, we can draw the conclusion that the linear regression model can fit the data well. In the regression model, we can see that the relationship between sales and cost on profit is the largest, providing sufficient decision-making basis for enterprise managers. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the flow chart of the present invention; arrows represent the direction of data or information.

[0027] Application Examples Let’s take the analysis of corporate financial data as an example to build an analysis model.

[0028] In order to demonstrate the data analysis method based on association rules and regression models proposed by the present invention, an analysis example is constructed by analyzing the financial data of a virtual enterprise. The following is an example template, which shows the analysis process in as much detail as possible.

[0029] 1. Data Acquisition First, obtain relevant financial data sets from the company's internal system, including the company's sales, costs, profits, and other data. Assume that a financial data set containing 2019 and 2020 has been obtained and saved in a CSV file.

[0030] Year, Month, Sales, Cost, Profit 2019,1,1000000,800000,200000 2019,2,1200000,900000,300000 2019,3,800000,600000,200000 2019,4,1100000,800000,300000 2019,5,900000,700000,200000 2019,6,1300000,950000,350000 2019,7,1100000,900000,200000 2019,8,1400000,1000000,400000 2019,9,1200000,900000,300000 2019,10,900000,600000,300000 2019,11,1000000,800000,200000 2019,12,1100000,900000,200000 2020,1,1300000,1000000,300000 2020,2,1500000,1100000,400000 2020,3,900000,700000,200000 2020,4,1200000,900000,300000 2020,5,1000000,750000,250000 2020,6,1600000,1200000,400000 2020,7,1300000,1000000,300000 2020,8,1700000,1300000,400000 2020,9,1400000,1100000,300000 2020,10,1000000,750000,250000 2020,11,1100000,800000,300000 2020,12,1200000,900000,300000 2. Data Preprocessing Before analyzing financial data, some data preprocessing operations are required to ensure the quality of the data. Use Python's pandas library to read and preprocess the data, including removing duplicate data and processing missing values. The code is as follows: import pandas as pd # Read the dataset df = pd.read_csv('financial_data.csv') # Remove duplicates df.drop_duplicates(inplace=True) # Handling missing values 3. Association Rule Mining Before mining association rules for financial data, the data needs to be converted and encoded. Normalize the numerical data in the data set and perform one-hot encoding on the categorical data. Then, use the Apriori algorithm in the mlxtend library of Python to mine association rules for the data set. The code is as follows: import numpy as np from sklearn.preprocessing import MinMaxScaler, OneHotEncoder from mlxtend.frequent_patterns import apriori, association_rules # Normalize numerical data scaler = MinMaxScaler() df[['Sales', 'Cost', 'Profit']] = scaler.fit_transform(df[['Sales', 'Cost', 'Profit']]) # One-hot encoding of categorical data encoder = OneHotEncoder(sparse=False) df_encoded = pd.DataFrame(encoder.fit_transform(df[['Year', 'Month']])) df_encoded.columns = encoder.get_feature_names(['Year', 'Month']) df.drop(['Year', 'Month'], axis=1, inplace=True) df = pd.concat([df_encoded, df], axis=1) # Association rule mining frequent_itemsets = apriori(df, min_support=0.1, use_colnames=True) rules = association_rules(frequent_itemsets, metric="lift", min_threshold=1) The results of association rule mining are as follows: antecedents consequents antecedent support consequent support supportconfidence lift leverage conviction 0 (Year_2019) (Profit) 0.5 0.583333 0.375 0.75 1.285714 0.0833331.666667 1 (Profit) (Year_2019) 0.583333 0.5 0.375 0.642857 1.285714 0.0833331.285714 2 (Sales) (Profit) 0.833333 0.583333 0.458333 0.55 0.944444 0.0277780.916667 3 (Cost) (Profit) 0.833333 0.583333 0.5 0.6 1.028571 0.0138891.083333 4 (Profit) (Sales) 0.583333 0.833333 0.458333 0.785714 0.9428570.027778 0.916667 5 (Profit) (Cost) 0.583333 0.833333 0.5 0.857143 1.028571 0.0138891.083333 According to the results of association rule mining, the following conclusions can be drawn: There is a strong correlation between sales and profits in 2019, which may be due to 2019 being a profitable year for the company; There is a certain correlation between sales and profits, and between costs and profits, but the degree of correlation is not high.

[0031] IV. Regression Analysis Before performing regression analysis on financial data, some data preprocessing operations are required, including dividing the data set into training sets and test sets, normalizing numerical data, etc. Then, the linear regression model in Python's scikit-learn library is used for regression analysis. The code is as follows: from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression # Split the dataset into training and testing sets X_train, X_test, y_train, y_test = train_test_split(df.drop('Profit',axis=1), df['Profit'], test_size=0.3, random_state=42) # Normalize numerical data scaler = MinMaxScaler() X_train[['Sales', 'Cost']] = scaler.fit_transform(X_train[['Sales', 'Cost']]) X_test[['Sales', 'Cost']] = scaler.transform(X_test[['Sales', 'Cost']]) # Linear regression model model = LinearRegression() model.fit(X_train, y_train) # Model evaluation train_score = model.score(X_train, y_train) test_score = model.score(X_test, y_test) print(f'Training score: {train_score:.2f}') print(f'Testing score: {test_score:.2f}') The results of the regression analysis are as follows: Training score: 0.82 Testing score: 0.77 According to the results of regression analysis, the following conclusions were obtained: The linear regression model fits the data well, with a training score of 0.82 and a test score of 0.77, which indicates that the model has good generalization ability. In the regression model, sales and costs have the greatest impact on profits.

[0032] V. Conclusion Through association rule mining and regression analysis of the financial data of virtual enterprises, it is found that there is a strong correlation between sales and profits in 2019; there is a certain correlation between sales and profits, and between costs and profits, but the degree of correlation is not high; sales and costs have the greatest impact on profits.

[0033] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A data analysis method based on association rules and regression models, characterized in that: include: Get data set: Get the data set to be analyzed from the data source; Data preprocessing: perform preprocessing operations such as cleaning, deduplication, missing value processing, and outlier processing on the data set to ensure data quality; Association rule mining: Use association rule mining algorithms such as the Apriori algorithm to discover the correlation between data by mining frequent item sets and association rules; Feature engineering: Extract features from data sets based on mined association rules and domain knowledge, including numerical features and categorical features; Establish a regression model: Use regression models such as linear regression and logistic regression to establish a relationship between features and target variables, and perform model training and evaluation; Model optimization: perform optimization operations such as parameter adjustment and feature selection on the established regression model to improve the prediction performance of the model; Model application: Use the optimized regression model to predict and analyze new data to provide decision support and business guidance.

2. The data analysis method based on association rules and regression model according to claim 1 is characterized in that: The data preprocessing steps include: Cleaning data: removing duplicate records, incomplete records, inconsistent records and other data problems to ensure data accuracy and consistency; Missing value processing: For data with missing values, mean filling, median filling, mode filling, regression filling and other methods are used to process them to ensure the integrity and availability of the data; Outlier processing: For data with outliers, use deletion, replacement, smoothing and other methods to ensure the validity and reliability of the data.

3. The data analysis method based on association rules and regression model according to claim 1 is characterized in that: The association rule mining steps include: Frequent item set mining: Use frequent item set mining algorithms such as the Apriori algorithm to discover frequently occurring combination relationships in the data and construct frequent item sets; Association rule mining: Use association rule mining algorithms to discover association rules in data, including strong rules and weak rules, and build association rule sets; Association rule screening: According to the support, confidence, and lift of association rules, the association rules with significance and practical value are screened out.

4. The data analysis method based on association rules and regression model according to claim 1 is characterized in that: The feature engineering step includes: Feature extraction: Extract features from data sets based on mined association rules and domain knowledge, including numerical features and classification features; Feature encoding: For categorical features, one-hot encoding, binary encoding and other methods are used to encode them and convert them into numerical features; Feature selection: Based on the relevance, importance and other indicators of the features, select the features that have a significant impact on the target variable to improve the prediction performance of the model.

5. The data analysis method based on association rules and regression model according to claim 1 is characterized in that: The steps of building a regression model include: Target variable selection: Select appropriate target variables based on business needs and data characteristics, such as continuous variables, binary categorical variables, multivariate categorical variables, etc.; Model selection: According to the type of target variable and data characteristics, select an appropriate regression model, such as linear regression, logistic regression, ridge regression, Lasso regression, etc.; Model training: Use the training data set to train the regression model, find the optimal model parameters, and improve the model's prediction performance; Model evaluation: Use the test data set to evaluate the established regression model and calculate the model's evaluation indicators, such as mean square error, mean absolute error, accuracy, etc.

6. The data analysis method based on association rules and regression model according to claim 1 is characterized in that: The model optimization steps include: Parameter adjustment: For the established regression model, the prediction performance of the model can be improved by adjusting the model parameters, such as learning rate, regularization coefficient, etc. Feature selection: Based on the relevance and importance of features, select features that have a significant impact on the target variable to improve the prediction performance of the model; Model fusion: Multiple regression models are integrated to improve the prediction performance and robustness of the model.

7. The data analysis method based on association rules and regression model according to claim 1 is characterized in that: The model application steps include: New data preprocessing: For new data to be analyzed, perform the same preprocessing operations as the training data to ensure the quality and availability of the data; Feature extraction: Extract features of new data based on existing association rules and feature engineering methods for model prediction and analysis; Model prediction: Use the optimized regression model to predict and analyze new data to obtain prediction results and prediction errors; Decision support: Provide decision support and business guidance based on forecast results and business needs, and provide reference and assistance to decision makers.

Images