Big data mining and analysis management system

Abstract

The present invention discloses a big data mining and analysis management system, which relates to the field of data mining technology and solves the technical problem of inaccurate mining caused by single analysis based on association relationship. The present invention effectively handles the problems of missing values, redundant data and the like in the data by preprocessing the data, improves the data quality, performs association analysis on the same type of preprocessed data, and can accurately discover the potential relationship between the data by calculating the association degree and judging the correlation coefficient. A mining model is established according to the associated data packets, and the performance and generalization ability of the model are improved through reasonable data set division, parameter tuning and model training evaluation. A neural network mining model is adopted and optimized in combination with forward propagation and back propagation algorithms, so that complex data relationships can be better handled. The mining and analysis using the model can improve the comprehensiveness of data mining.

Description

Technical Field

[0001] The present invention relates to the field of data mining technology, and in particular to a big data mining analysis and management system. Background Art

[0002] In today's big data era, the value of data is becoming increasingly prominent. Enterprises and organizations need to obtain a large amount of data from various sources, including internal databases, external data sources, sensor data, and social media data, to support decision-making, business optimization, and innovative development. However, these data often have quality problems, such as missing values, outliers, and data inconsistencies, and need to be preprocessed for effective analysis and mining.

[0003] According to the publication number CN116578613B, a data mining system for big data analysis is disclosed, including: a data preprocessing module, which identifies entities contained in government data based on government data, generates a government data map based on the relationship between entities, and generates entity vectors for entities; a graph generation module, which generates a graph matrix based on the government data map; a region generation module, which divides the graph matrix equally into N non-overlapping sub-regions; sequentially splices the region parameters of all sub-regions to generate sub-region vectors; a data processing module, which is used to input the graph matrix, sub-region vectors and entity vectors into a data encoding model, and output classification labels related to the data mining target.

[0004] The above patent learns the graph matrix and sampling area parameters to make up for the missing associations between entities, which is suitable for data mining processing of government big data. However, with the above technical means, there is an incomplete analysis when analyzing the data. At the same time, the single analysis based on association may lead to inaccurate mining. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a big data mining and analysis management system, which solves the problem of inaccurate mining caused by single analysis based on association relationships.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a big data mining and analysis management system, comprising:

[0007] A preprocessing module is used to perform data preprocessing on the acquired relevant data, perform data cleaning on the relevant data to obtain cleaned data, perform data integration on the cleaned data to obtain integrated data, then perform standard transformation on the integrated data to obtain preprocessed data, and transmit the obtained preprocessed data to the correlation calculation module;

[0008] The association calculation module is used to analyze the acquired pre-processed data, classify the acquired pre-processed data into the same type to obtain the same type of pre-processed data, analyze the association between the same type of pre-processed data, obtain the corresponding association data packet, and transmit the association data packet to the mining model establishment module;

[0009] The mining model establishment module is used to establish the corresponding mining model according to the obtained associated data packets, divide the obtained associated data packets into data sets, select standard model parameters according to the selected mining technology, train the model with the training set, evaluate the trained model with the test set to obtain the mining model, and use the standard model preservation format, and at the same time transmit the mining model to the mining information output module.

[0010] As a further solution of the present invention: it also includes a data acquisition module and a mining information output module;

[0011] A data acquisition module is used to acquire relevant data and transmit the acquired relevant data to a preprocessing module;

[0012] The mining information output module is used to store the mining model.

[0013] As a further solution of the present invention: the specific method in which the preprocessing module preprocesses the relevant data is:

[0014] All relevant data are obtained and labeled as i, where i=1, 2, ..., j, where j represents the number of relevant data. Then, relevant data with missing values ​​in relevant data i are extracted and recorded as missing data, and the missing data are filled.

[0015] As a further solution of the present invention: the specific method of the preprocessing module to fill the missing data is:

[0016] Get all missing data, extract non-missing values ​​from the missing data, and sort the extracted non-missing values. If the number of non-missing values ​​is an odd number, the median is the middle value after sorting. If the number of non-missing values ​​is an even number, the median is the average of the two middle values.

[0017] Use the calculated median to fill the missing values ​​in the variable, traverse each data point of the variable with missing values ​​in the data set, and if the data point is a missing value, replace the missing value with the previously calculated median;

[0018] And so on, we get the cleaned data.

[0019] As a further solution of the present invention: the specific method in which the preprocessing module integrates the cleaned data to obtain the integrated data is:

[0020] Acquire all cleaned data and integrate the cleaned data from multiple data sources. The specific integration method is: merge based on common key fields, and deduplicate the redundant data in the cleaned data to obtain integrated data.

[0021] As a further solution of the present invention: the specific method in which the preprocessing module standardizes the integrated data to obtain the preprocessed data is:

[0022] Obtain all integrated data, quantify the integrated data, and substitute the obtained quantified integrated data into the formula The preprocessed data is obtained by standardization, and X is the original quantitative integrated data. is the mean of the data, is the standard deviation of the data.

[0023] As a further solution of the present invention: the specific manner in which the association calculation module analyzes the pre-processed data is:

[0024] Obtain all preprocessed data, denoted as n, where n=1, 2, ..., m, where m represents the number of preprocessed data, then obtain the data type corresponding to the preprocessed data n, and classify the preprocessed data into the same type according to the data type to obtain preprocessed data of the same type;

[0025] Analyze the correlation between the same type of preprocessed data, obtain the sub-data in the same type of preprocessed data, and label them as a, where a=1, 2, ..., b, where b represents the number of sub-data, then obtain the sub-data corresponding to another group of the same type of preprocessed data, and label them as o, where o=1, 2, ..., p, where p represents the number of another group of sub-data, then calculate the correlation between the sub-data a labeled 1 and the sub-data o labeled 1, as follows: in is the covariance of a1 and o1, and are the standard deviations of sub-data a1 and o1 respectively. The Pearson correlation coefficients corresponding to sub-data a1 and o1 are obtained by the above formula, and the Pearson correlation coefficients are judged at the same time.

[0026] As a further solution of the present invention: the specific manner in which the association calculation module determines the Pearson correlation coefficient is:

[0027] If the Pearson correlation coefficient is -1, it means that the sub-data a1 and o1 are completely negatively correlated. If the Pearson correlation coefficient is 1, it means that the sub-data a1 and o1 are completely positively correlated. If the Pearson correlation coefficient is 0, it means that the sub-data a1 and o1 are not correlated. And so on, the correlation analysis is performed according to the permutation and combination method.

[0028] Then, the number of data with correlation is obtained, and the ratio value is calculated. At the same time, the ratio value is compared with the preset value. If the ratio value is greater than the preset value, it means that the corresponding two groups of pre-processed data of the same type are correlated, and the two are bundled to obtain a correlated data packet. If the ratio value is less than the preset value, it means that the corresponding two groups of pre-processed data of the same type are not correlated and are not processed.

[0029] By analogy, all data of the same type are processed in the same way, and corresponding associated data packets are obtained, and the associated data packets are transmitted to the mining model building module.

[0030] As a further solution of the present invention: the specific method of the mining model building module to build the corresponding mining model is:

[0031] Obtain all associated data including, and label them as k, where k=1, 2, ..., h, and divide the associated data packets into training sets and test sets according to the preset ratio of the total number;

[0032] Then, the obtained training set and test set are verified and analyzed to determine whether the training set and the test set have different categories or standard distribution characteristics. If not, they are not processed. If so, they are subjected to secondary division processing. The specific method of secondary division is: according to the characteristics of the associated data packets, one or more variables are selected as the basis for stratification. For each stratification, the associated data packets are randomly selected according to a predetermined ratio as the training set and the test set.

[0033] Grid search is used to select standard model parameters. According to the nature of the associated data packets and the characteristics of the data, a standard mining model is selected. The model is trained using a training set. The prepared training set feature data is input into the model. The model learns and optimizes based on the input data and target data.

[0034] Use the test set to evaluate the trained model, and use the mean square error to optimize the mining model. Use the standard model saving format to save the model parameters and generate the mining model.

[0035] The present invention provides a big data mining and analysis management system. Compared with the prior art, it has the following beneficial effects:

[0036] The present invention effectively handles the problems of missing values, redundant data, etc. in the data by preprocessing the data, improves the data quality, performs correlation analysis on the same type of preprocessed data, and can accurately find the potential relationship between the data by calculating the correlation degree and judging the correlation coefficient. The mining model is established according to the associated data packet, and the performance and generalization ability of the model are improved through reasonable data set division, parameter tuning and model training evaluation. The neural network mining model is adopted, and is optimized in combination with the forward propagation and back propagation algorithms, which can better handle complex data relationships, and the mining analysis using the model can improve the comprehensiveness of data mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a block diagram of the system principle of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 The present application provides a big data mining, analysis and management system, including: a data acquisition module, a preprocessing module, an association calculation module, a mining model building module and a mining information output module. At the same time, it can be seen from the accompanying drawings that the corresponding functional modules are unidirectionally electrically connected.

[0040] The data acquisition module is used to acquire relevant data and transmit the acquired relevant data to the preprocessing module. The specific acquired relevant data includes internal databases, external data sources (such as industry reports, public data sets, etc.), sensor data, social media data, etc.

[0041] The preprocessing module is used to preprocess the acquired relevant data. First, the relevant data is cleaned to obtain cleaned data. The specific method of data cleaning is as follows:

[0042] Get all the relevant data, and label the relevant data as i, where i=1, 2, ..., j, where j represents the number of relevant data. Then extract the relevant data with missing values ​​in the relevant data i and record them as missing data. At the same time, fill the missing data, and the specific filling method is median filling (calculate the median of its non-missing values). The specific steps of median filling are as follows:

[0043] Get all missing data, extract non-missing values ​​from the missing data, and sort the extracted non-missing values. If the number of non-missing values ​​is an odd number, the median is the middle value after sorting. If the number of non-missing values ​​is an even number, the median is usually the average of the two middle values.

[0044] For example, for the data set [1,3,5,7,9], the median is 5; for the data set [2,4,6,8], the median is (4+6) / 2=5.

[0045] Use the calculated median to fill the missing values ​​in the variable, traverse each data point of the variable with missing values ​​in the data set, if the data point is a missing value, replace the missing value with the previously calculated median, and so on, to obtain the cleaned data;

[0046] Then, data integration is performed on the cleaned data to obtain integrated data, and the specific method of data integration is as follows:

[0047] Obtain all clean data and integrate the clean data from multiple data sources. The specific integration method is: merge based on common key fields, and remove redundant data in the clean data to obtain integrated data;

[0048] Finally, the integrated data is transformed into preprocessed data by standard conversion, and the preprocessed data is transmitted to the correlation calculation module. The specific standard conversion method is:

[0049] Obtain all integrated data, quantify the integrated data, and substitute the obtained quantified integrated data into the formula Standardization is performed to obtain preprocessed data, where X is the original quantitative integrated data, is the mean of the data, is the standard deviation of the data.

[0050] The association calculation module is used to analyze the acquired pre-processed data. First, the acquired pre-processed data is classified into the same type to obtain the same type of pre-processed data. The specific classification method is as follows:

[0051] Obtain all preprocessed data, record them as n, and n=1, 2, ..., m, where m represents the number of preprocessed data, then obtain the data type corresponding to the preprocessed data n, and classify the preprocessed data into the same type according to the data type, and finally obtain the preprocessed data of the same type;

[0052] Next, the correlation between the same type of preprocessed data is analyzed, specifically by calculating the correlation between the same type of preprocessed data. The specific calculation method is as follows:

[0053] Obtain sub-data from the same type of pre-processed data. For example, the same type of pre-processed data obtained here is marked as A, and the sub-data here represents the pre-processed data of the corresponding type. For example, the same type of pre-processed data includes text data, image data and audio data, and the corresponding sub-data is all the pre-processed data in the text data, and is marked as a, and a=1, 2, ..., b, where b represents the number of sub-data. Then obtain the sub-data corresponding to another group of pre-processed data of the same type. For example, the same type of pre-processed data obtained here is marked as B, and is marked as o, and o=1, 2, ..., p, where p represents the number of another group of sub-data. Then calculate the degree of association between the sub-data a marked as 1 and the sub-data o marked as 1. Specifically, analyze the degree of association between the sub-data in A and the sub-data in B. The specific calculation method is as follows: in is the covariance of a1 and o1, and are the standard deviations of sub-data a1 and o1 respectively. The Pearson correlation coefficients corresponding to sub-data a1 and o1 are obtained by the above formula, and the Pearson correlation coefficients are judged at the same time;

[0054] If the Pearson correlation coefficient takes the value of -1, it means that the sub-data a1 and o1 are completely negatively correlated. If the Pearson correlation coefficient takes the value of 1, it means that the sub-data a1 and o1 are completely positively correlated. If the Pearson correlation coefficient takes the value of 0, it means that the sub-data a1 and o1 are not correlated. And so on, the correlation analysis is performed in the form of permutations and combinations. Specifically, the permutation and combination method here is expressed as follows: first, the sub-data a1 labeled 1 in A is respectively subjected to correlation analysis with the sub-data o in B, and the o label takes the value of 1, 2, ..., p, and then the sub-data a2 labeled 2 is respectively subjected to correlation analysis with the sub-data o.

[0055] Then, the number of associated data is obtained, and the number of associated data here means the number of associated data in A and B, and the number ratio is calculated, and the number ratio is compared with the preset value, and the specific value of the preset value here is set by the operator. If the number ratio is greater than the preset value, it means that the corresponding two groups of pre-processed data of the same type are associated, and the two are bundled to obtain an associated data packet. If the number ratio is less than the preset value, it means that the corresponding two groups of pre-processed data of the same type are not associated and are not processed;

[0056] Similarly, all data of the same type are processed in the same way and corresponding associated data packets are obtained. Here, the number of associated data packets is two. If there is a group of preprocessed data of the same type associated with multiple groups of preprocessed data of other types, two groups are also bundled and processed, and the associated data packets are transmitted to the mining model building module at the same time.

[0057] The mining model building module is used to build a corresponding mining model according to the acquired associated data package, and building the mining model specifically includes the following steps:

[0058] In the first step, the acquired associated data packets are divided into data sets, all associated data are obtained, and are labeled as k, where k=1, 2, ..., h. At the same time, the associated data packets are divided into training sets and test sets according to a preset ratio according to the total number, and the preset ratio here is set by the operator. In this application, the preset ratio is 7:3, and the selection is performed by random sampling. For example, assuming that there are 1000 associated data packets, 700 samples are randomly selected as training sets according to the ratio of 7:3, and the remaining 300 samples are used as test sets.

[0059] Then, the obtained training set and test set are verified and analyzed to determine whether the training set and the test set have different categories or standard distribution characteristics. If not, they will not be processed. If so, they will be divided twice. The specific method of the secondary division is: according to the characteristics of the associated data packets, one or more variables are selected as the basis for stratification. For each stratification, the associated data packets are randomly selected as the training set and the test set according to a predetermined ratio.

[0060] In the second step, according to the selected mining technology, standard model parameters are selected, and parameter tuning methods such as grid search or random search are adopted. Specifically, grid search is an exhaustive search method, which traverses all possible combinations within a given parameter range, performs model training and evaluation on each combination, and finally selects the parameter combination with the best performance. Random search randomly selects parameter combinations within a given parameter range for testing, which is more efficient than grid search, especially when the parameter space is large. According to the properties of the associated data packets and the characteristics of the data, a standard mining model such as decision tree, neural network, support vector machine, etc. is selected. In this application, a neural network mining model is adopted. The neural network calculates the output through forward propagation, and then updates the weights and biases according to the loss function and back propagation algorithm.

[0061] The third step is to train the model using the training set, divide the training set data into features (input variables) and targets (output variables), set the corresponding number of layers, number of neurons, learning rate and other parameters based on the neural network model, input the prepared training set feature data into the model, and the model learns and optimizes based on the input data and target data. The specific optimization includes initializing the model parameters, calculating the error between the model's predicted value and the actual target value, adjusting the model parameters based on the error, and repeating the above steps until certain stopping conditions are met.

[0062] The fourth step is to use the test set to evaluate the trained model, and the evaluation indicators can include accuracy, recall rate, F1 value, mean square error, etc., and optimize the mining model. Repeat the above steps to obtain the mining model.

[0063] Step 5: When the model training is completed and satisfactory performance is achieved, save the model parameters. You can use standard model saving formats, such as pickle, HDF5, etc. to save the model as a file. At the same time, you can also record relevant information about the model, such as model structure, parameter settings, statistical information of training data, etc., so that you can reproduce and analyze it when needed.

[0064] In a specific example, for example, suppose we use Python's scikit-learn library to train a random forest classifier for image classification tasks. When the model training is completed and performs well on the test set, we can use the pickle module to save the model as a file, and the specific saving code is as follows:

[0065] importpickle

[0066] #Trained random forest model

[0067] model=RandomForestClassifier()

[0068] #Save model

[0069] withopen('random_forest_model.pkl','wb')asf:

[0070] pickle.dump(model,f)

[0071] #At the same time, you can save information such as model structure and parameter settings into a text file

[0072] withopen('model_info.txt','w')asf:

[0073] f.write(f"Model type: Random Forest Classifier\n")

[0074] f.write(f"Parameter settings: n_estimators={model.n_estimators}, max_depth={model.max_depth}, etc.\n")

[0075] f.write(f"Training data statistics: mean = {training_data_mean}, standard deviation = {training_data_std}, etc.").

[0076] Then the established mining model is transmitted to the mining information output module.

[0077] Mining information output module, which is used to store the mining model.

[0078] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0079] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. Big data mining and analysis management system, characterized by: include: The preprocessing module is used to perform data preprocessing on the acquired relevant data, perform data cleaning on the relevant data to obtain cleaned data, and perform data integration on the cleaned data to obtain integrated data, and then perform standard transformation on the integrated data to obtain preprocessed data, and transmit the obtained preprocessed data to the associated calculation module. The specific method of analyzing the preprocessed data is as follows: Obtain all preprocessed data and record them as n, where n=1, 2, ..., m, where m represents the number of preprocessed data, then obtain the data type corresponding to the preprocessed data n, and classify the preprocessed data into the same type according to the data type to obtain preprocessed data of the same type; Analyze the correlation between the same type of preprocessed data, obtain the sub-data in the same type of preprocessed data, and label them as a, where a=1, 2, ..., b, where b represents the number of sub-data, then obtain the sub-data corresponding to another group of the same type of preprocessed data, and label them as o, where o=1, 2, ..., p, where p represents the number of another group of sub-data, then calculate the correlation between the sub-data a labeled 1 and the sub-data o labeled 1, the formula is as follows: in is the covariance of a1 and o1, and are the standard deviations of sub-data a1 and o1 respectively. The Pearson correlation coefficients corresponding to sub-data a1 and o1 are obtained by the above formula, and the Pearson correlation coefficients are judged at the same time; If the Pearson correlation coefficient is -1, it means that the sub-data a1 and o1 are completely negatively correlated. If the Pearson correlation coefficient is 1, it means that the sub-data a1 and o1 are completely positively correlated. If the Pearson correlation coefficient is 0, it means that the sub-data a1 and o1 are not correlated. And so on, the correlation analysis is performed according to the permutation and combination method. Then, the number of data with correlation is obtained, and the ratio value is calculated. At the same time, the ratio value is compared with the preset value. If the ratio value is greater than the preset value, it means that the corresponding two groups of pre-processed data of the same type are correlated, and the two are bundled to obtain a correlated data packet. If the ratio value is less than the preset value, it means that the corresponding two groups of pre-processed data of the same type are not correlated and are not processed. By analogy, all data of the same type are processed in the same way, and the corresponding associated data packets are obtained, and the associated data packets are transmitted to the mining model building module; The association calculation module is used to analyze the acquired pre-processed data, classify the acquired pre-processed data into the same type to obtain the same type of pre-processed data, analyze the association between the same type of pre-processed data, obtain the corresponding association data packet, and transmit the association data packet to the mining model establishment module; The mining model building module is used to build a corresponding mining model according to the acquired associated data packets, divide the acquired associated data packets into data sets, select standard model parameters according to the selected mining technology, train the model with the training set, evaluate the trained model with the test set to obtain the mining model, and use the standard model saving format, and transmit the mining model to the mining information output module at the same time. The specific method of building the corresponding mining model is as follows: Obtain all associated data including, and label them as k, where k=1, 2, ..., h, and divide the associated data packets into training sets and test sets according to the preset ratio of the total number; Then, the obtained training set and test set are verified and analyzed to determine whether the training set and the test set have different categories or standard distribution characteristics. If not, they are not processed. If so, they are subjected to secondary division processing. The specific method of secondary division is: according to the characteristics of the associated data packets, one or more variables are selected as the basis for stratification. For each stratification, the associated data packets are randomly selected according to a predetermined ratio as the training set and the test set. Grid search is used to select standard model parameters. According to the nature of the associated data packets and the characteristics of the data, a standard mining model is selected. The model is trained using a training set. The prepared training set feature data is input into the model. The model learns and optimizes based on the input data and target data. Use the test set to evaluate the trained model, and use the mean square error to optimize the mining model. Use the standard model saving format to save the model parameters and generate the mining model.

2. The big data mining and analysis management system according to claim 1 is characterized in that: Also includes: Data acquisition module and mining information output module; A data acquisition module is used to acquire relevant data and transmit the acquired relevant data to a preprocessing module; The mining information output module is used to store the mining model.

3. The big data mining and analysis management system according to claim 1 is characterized in that: The specific method in which the preprocessing module preprocesses the relevant data is as follows: All relevant data are obtained and labeled as i, where i=1, 2, ..., j, where j represents the number of relevant data. Then, relevant data with missing values ​​in relevant data i are extracted and recorded as missing data, and the missing data are filled.

4. The big data mining and analysis management system according to claim 3 is characterized in that: The specific method of filling missing data by the preprocessing module is: Get all missing data, extract non-missing values ​​from the missing data, and sort the extracted non-missing values. If the number of non-missing values ​​is an odd number, the median is the middle value after sorting. If the number of non-missing values ​​is an even number, the median is the average of the two middle values. Traverse each data point of the variable with missing values ​​in the data set. If the data point is a missing value, replace the missing value with the median calculated previously; And so on, we get the cleaned data.

5. The big data mining and analysis management system according to claim 1 is characterized in that: The specific method in which the preprocessing module integrates the cleaned data to obtain the integrated data is: Acquire all cleaned data and integrate the cleaned data from multiple data sources. The specific integration method is: merge based on common key fields, and deduplicate the redundant data in the cleaned data to obtain integrated data.

6. The big data mining and analysis management system according to claim 1 is characterized in that: The specific method in which the preprocessing module standardizes the integrated data to obtain the preprocessed data is: Obtain all integrated data, quantify the integrated data, and substitute the obtained quantified integrated data into the formula The preprocessed data is obtained by standardization, and X is the original quantitative integrated data. is the mean of the data, is the standard deviation of the data.

Images