A big data prediction system based on machine learning

Abstract

The present invention discloses a big data prediction system based on machine learning, which relates to the technical fields of machine learning and big data. The system includes a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module, and a prediction output module. In the present invention, through the feature engineering module, specific algorithms are used, correlation is considered and a penalty factor is introduced to focus on relevant feature subsets to improve accuracy. The hidden layer and parameters of the model module can be adaptively adjusted. For example, high accuracy can be obtained in power load prediction. The data acquisition module updates and processes the format in real time to ensure the timeliness and availability of data. The preprocessing module uses a clustering algorithm to process outliers, and after determining the parameters, it corrects and normalizes them to meet the algorithm requirements, improving stability and accuracy. The combined action of these characteristics enables the system to perform excellently in prediction tasks, laying a solid foundation for accurate prediction.

Description

Technical Field

[0001] The present invention relates to the technical fields of machine learning and big data, and particularly to a big data prediction system based on machine learning. Background Art

[0002] With the wide application of big data and machine learning technologies in various fields, a series of defects have gradually emerged in traditional prediction systems.

[0003] In terms of feature engineering, most existing systems do not fully consider the correlation between features and lack effective means to avoid the selection of redundant features. This not only makes the model structure complex but also severely restricts the prediction accuracy.

[0004] In terms of the model structure, the number of hidden layers in common prediction models is usually fixed. This fixed structure cannot be adaptively adjusted according to the complexity of the data and the difficulty of the prediction task, making it difficult to well adapt to various different data characteristics and having an adverse impact on the prediction effect.

[0005] In the data acquisition module, many systems have obvious deficiencies. On the one hand, there is a lack of an effective real-time data update mechanism, and the latest big data samples cannot be obtained in a timely manner, resulting in poor data timeliness. On the other hand, for newly added data, either there is no format verification or it cannot be properly processed when the format does not meet the requirements, which directly affects the integrity and availability of the data and thus hinders the subsequent prediction work.

[0006] There are also problems in the data preprocessing link. In dealing with outliers, most systems do not determine appropriate parameters according to the characteristics of the data itself, and the processing methods are not scientific and reasonable enough. And there are also deficiencies in data normalization processing, making it difficult for the data to meet the strict requirements of machine learning algorithms, ultimately resulting in a decline in the stability and accuracy of the model. Summary of the Invention

[0007] The purpose of the present invention is to make up for the deficiencies of the existing technology and provide a big data prediction system based on machine learning. It can use specific algorithms through the feature engineering module, consider the correlation and introduce a penalty factor to focus on relevant feature subsets to improve the accuracy. The hidden layers and parameters of the model module can be adaptively adjusted, such as obtaining high accuracy in power load prediction. The data acquisition module updates and processes the format in real time to ensure data timeliness and availability. The preprocessing module uses a clustering algorithm to process outliers, corrects and normalizes after determining the parameters to meet the algorithm requirements and improve stability and accuracy. These characteristics work together to make the system perform excellently in prediction tasks and lay a solid foundation for accurate prediction.

[0008] To solve the above technical problems, the present invention provides the following technical solution: A big data prediction system based on machine learning, the system includes a data collection module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module and a prediction output module;

[0009] The data collection module is used to collect a big data sample set, the big data sample set covers historical data of multiple time nodes, and the data of each time node has multiple feature variables;

[0010] The data preprocessing module performs cleaning and normalization processing on the collected data, eliminates outliers and noise, and makes the data meet the input requirements of machine learning algorithms;

[0011] The feature engineering module extracts effective features from the preprocessed data, and uses a feature selection algorithm based on information gain ratio to select a feature subset with high correlation with the prediction target;

[0012] The machine learning prediction model module, the machine learning prediction model is based on an improved deep neural network architecture, includes an input layer, multiple hidden layers and an output layer, and the hidden layer adopts a mixed activation function and an adaptive weight adjustment mechanism;

[0013] The mixed activation function is f(x)=α×ReLU(x)+(1 - α)×Sigmoid(x), where x is the neuron input, α is the weight coefficient, and its value is determined by a dynamic adjustment strategy based on the data distribution during the training process. The initial value is set to 0.5, and as the number of training iterations increases, it is dynamically adjusted according to the mean and variance of each batch of data. The adjustment formula is α t+1 =α t +β×(μ - σ), where α t is the α value at the t-th iteration, β is the adjustment step size, and its value range is [0.01, 0.1], μ is the mean of the current batch of data, and σ is the variance of the current batch of data;

[0014] The adaptive weight adjustment mechanism adopts a comprehensive evaluation method based on gradient information and neuron activity. For the weight of the i-th neuron in the l-th layer Its update formula is where is the weight value at the t-th iteration, η is the learning rate, and its value range is [0.001, 0.1], is the gradient of the loss function E with respect to this weight, γ is the activity adjustment coefficient, and its value range is [0.005, 0.05], is the activity of this neuron, and the calculation formula is where T is the total number of iterations, is the activation value of this neuron in the l-th layer at the t-th iteration;

[0015] The model training module uses the processed feature subset to train the innovative machine learning prediction model and optimizes it using a custom loss function. The loss function is where N is the number of samples, y i is the true value of the i-th sample, is the model prediction value, λ is the regularization parameter, and its value range is [0.0001, 0.001]. M is the total number of model weights, and w j is the j-th weight;

[0016] The prediction output module inputs the data to be predicted into the trained model to obtain the prediction result.

[0017] Furthermore, the data acquisition module further includes a real-time data update sub-module. This sub-module obtains the latest big data samples in real time through regular connection and data synchronization mechanisms with the data source to ensure the timeliness of the data. For newly added data, data format verification is first performed. If the format does not meet the requirements, a format conversion tool is automatically called for conversion, and then it is incorporated into the data cleaning and normalization processing flow.

[0018] Even further, the outlier processing in the data preprocessing module adopts a density-based outlier detection algorithm for clustering. The specific steps are as follows:

[0019] Step 1: Perform clustering analysis on the data. Use the density-based clustering algorithm DBSCAN to divide the data points into different clusters. The neighborhood radius Eps and the minimum number of samples MinPts of the core points are adaptively determined according to the dimensionality and distribution characteristics of the data. The optimal parameter combination is found through multiple experiments and evaluations.

[0020] Step 2: Mark the points that do not belong to any cluster as outliers, that is, abnormal values. For the detected outliers, according to their distances from adjacent data points and the data distribution, linear interpolation or neighboring value replacement methods are used for correction.

[0021] Even further, in the feature selection algorithm based on information gain ratio in the feature engineering module, when calculating the information gain ratio, the correlation between features is considered. For feature groups with high correlation, a penalty factor is introduced to reduce their information gain ratio, thus avoiding the selection of redundant features;

[0022] The specific calculation formula is: where GainRatio(F) is the information gain ratio of feature F, Gain(F) is the information gain of feature F, SplitInfo(F) is the split information value of feature F, ρ is the penalty coefficient, with a value range of [0.1, 0.5], Corr(F,F i ) is the correlation coefficient between feature F and the i-th other feature, and n is the number of features related to feature F.

[0023] Furthermore, the number of hidden layers in the machine learning prediction model module is automatically adjusted according to the complexity of the data and the difficulty of the prediction task, and the adjustment strategy is based on the performance evaluation metrics of the model on the validation set;

[0024] Specifically, when the loss function of the model on the validation set no longer decreases for K consecutive iterations, add a hidden layer, and at the same time adjust the number of neurons in each layer accordingly. The initial value of the number of neurons in the newly added hidden layer is p times the number of neurons in the previous layer, where p ranges from [0.5, 1.5], and is fine-tuned according to the performance during subsequent training. When overfitting occurs in the model on the validation set, reduce a hidden layer, and re-initialize and optimize the training of the neuron connection weights in the remaining hidden layers.

[0025] Furthermore, the learning rate η in the model training module adopts an adaptive adjustment strategy and is dynamically updated according to the gradient change of the model during training;

[0026] The specific formula is: where η t is the learning rate at the t-th iteration, is the gradient of the loss function at the t-th iteration, is the initial gradient of the loss function, ∈ is a very small positive number to prevent the denominator from being zero, with a value of 1e -8 , σ t is a smoothing factor based on the gradient history information, and the calculation formula is where α is the smoothing coefficient, with a value range of [0.7, 0.9].

[0027] Furthermore, the regularization parameter λ used in the model training module is determined by cross-validation on the training set to find its optimal value;

[0028] The specific operation is as follows: divide the training set into M subsets of equal size. Each time, select M - 1 of these subsets as the training subset, and the remaining one subset as the validation subset. For different candidate sets of λ values {λ1, λ2, …, λ n}, train the model on each training subset and evaluate the model performance on the corresponding validation subset. Use the mean absolute error (MAE) as the evaluation metric, and select the λ value that minimizes the MAE on the validation set as the final regularization parameter for model training.

[0029] Furthermore, the prediction output module further includes a result visualization sub-module, which presents the prediction results to the user in an intuitive chart form, including various forms such as line charts, bar charts, and scatter plots. The user can select different chart types for viewing according to their needs;

[0030] At the same time, provide a detailed analysis report of the prediction results, including the comparison between the predicted value and the true value, error analysis, and trend analysis, to help users better understand the prediction results and data characteristics.

[0031] Furthermore, the system also includes a system performance evaluation module that regularly evaluates the prediction performance of the system and comprehensively evaluates it using the root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R 2 )

[0032] According to the evaluation results, generate a performance report and provide optimization suggestions. If it is found that the system performance deteriorates or an anomaly occurs, automatically trigger the model retraining or parameter adjustment mechanism to ensure that the system always maintains good prediction performance.

[0033] Compared with the prior art, the big data prediction system based on machine learning has the following beneficial effects:

[0034] First, the feature engineering module of the present invention uses a feature selection algorithm based on the information gain ratio, considers the correlation between features, and avoids selecting redundant features by introducing a penalty factor. This enables the model to focus on the feature subset highly relevant to the prediction target, thereby improving the prediction accuracy of the model. The number of hidden layers of the machine learning prediction model module can be automatically adjusted according to the data complexity and the difficulty of the prediction task. At the same time, parameters such as the weight coefficient α of the mixed activation function, the learning rate η, and the activity adjustment coefficient γ in the adaptive weight adjustment mechanism can all be dynamically adjusted. In the power load prediction embodiment, this adaptive adjustment mechanism enables the model to better adapt to the characteristics of power load data and finally obtain a high prediction accuracy.

[0035] Second, the real-time data update sub-module of the data acquisition module of the present invention can obtain the latest big data samples in real time through a regular connection and data synchronization mechanism, ensuring the timeliness of the data. For newly added data, the data format is first verified. If the format does not meet the requirements, the format conversion tool is automatically called for conversion, and the data format is effectively processed, ensuring the integrity and availability of the data, laying a foundation for subsequent accurate prediction. The data preprocessing module uses a clustering-based outlier detection algorithm to process outliers, determines appropriate parameters according to the data characteristics, and then uses methods such as linear interpolation or neighboring value replacement for correction. After that, all data is normalized. In this way, outliers are effectively processed, making the data meet the input requirements of machine learning algorithms and improving the stability and accuracy of the model.

[0036] Other advantages, objectives, and features of the present invention will be described to some extent in the subsequent specification, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0038] Figure 1 It is a flowchart of a big data prediction system based on machine learning. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0040] Embodiment 1: Stock Market Price Prediction

[0041] In the stock market, predicting the trend of stock prices is crucial for investors. The big data prediction system based on machine learning of the present application is used to predict the price of a certain stock in the future for a period of time.

[0042] Data collection and preprocessing include a data collection module and a data preprocessing module;

[0043] Data Acquisition Module: Collect historical trading data of the stock from the stock trading database, including multiple feature variables such as opening price, closing price, highest price, lowest price, trading volume, etc. The time span is the past 5 years, covering multiple time nodes, with each trading day as a time node. At the same time, through the real-time data update sub-module, connect to the data source and synchronize data every 15 minutes to obtain the latest trading data. For newly added data, first verify whether its format meets the requirements. If it does not meet the requirements, call the format conversion tool to convert it into a format that the system can process, and then incorporate it into the data cleaning and normalization process.

[0044] Data Preprocessing Module: Use the density-based spatial clustering of applications with noise (DBSCAN) algorithm to handle outliers. According to the dimensionality and distribution characteristics of the data, determine that the neighborhood radius Eps of the core point is 0.5, and the minimum number of samples MinPts is 5. Mark the points that do not belong to any cluster as outliers, and for these outliers, use the method of linear interpolation to correct them according to their distances from adjacent data points and the data distribution. Then normalize all the data to meet the input requirements of machine learning algorithms.

[0045] Feature Engineering:

[0046] Feature Engineering Module: Extract effective features from the preprocessed data, and use the feature selection algorithm based on information gain ratio to select a feature subset that is highly relevant to the prediction target. When calculating the information gain ratio, consider the correlation between features, and set the penalty coefficient ρ to 0.3. The finally selected feature subset includes the closing price and trading volume of the past 10 trading days, as well as the price volatility of the past 5 trading days, etc.

[0047] Model Construction and Training include the Machine Learning Prediction Model Module and the Model Training Module;

[0048] Machine Learning Prediction Model Module: Build a prediction model based on an improved deep neural network architecture, including an input layer, 3 hidden layers, and an output layer. The hidden layer uses a mixed activation function and an adaptive weight adjustment mechanism. In the mixed activation function, the initial value of the weight coefficient α is 0.5, and the adjustment step size β is set to 0.05. In the adaptive weight adjustment mechanism, the initial value of the learning rate η is 0.01, and the activity adjustment coefficient γ is set to 0.02;

[0049] Model training module: Use the processed feature subset to train the model, and optimize it using a custom loss function. In the loss function, the number of samples N is the number of trading days in the past 5 years (about 1250). The regularization parameter λ is determined by cross-validation. The training set is divided into 5 subsets of equal size. Each time, 4 of these subsets are selected as the training subsets, and the remaining one subset is used as the validation subset. For different candidate sets of λ values (0.0001, 0.0003, 0.0005, 0.0007, 0.0009), train the model on each training subset and evaluate the model performance on the corresponding validation subset. Use the mean absolute error (MAE) as the evaluation metric, and finally determine that the value of λ is 0.0005.

[0050] Prediction and effect monitoring:

[0051] Prediction output module: Input the stock data to be predicted into the trained model to obtain the prediction result. The result visualization sub-module displays the prediction result to the user in the form of a line chart, and at the same time provides a detailed analysis report of the prediction result. The report includes the comparison between the predicted value and the true value, error analysis, trend analysis, etc. Monitor the prediction performance through the root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R 2 ) In this embodiment, after a period of monitoring, the RMSE is about 0.05, the MAE is about 0.03, and is about 0.85, indicating that the model has good prediction performance.

[0052] Embodiment 2: Power load prediction

[0053] In the power system, accurately predicting the power load is crucial for reasonably arranging the power generation plan and ensuring the stability of power supply. Use this system to predict the power load in a certain area for the next week.

[0054] Data collection and preprocessing include a data collection module and a data preprocessing module;

[0055] Data collection module: Collect the power load data and related meteorological data (such as temperature, humidity, wind speed, etc.) in this area for the past 3 years, with the time node being every hour. At the same time, through the real-time data update sub-module, connect to the data source and synchronize the data every 1 hour to obtain the latest power load and meteorological data. For the newly added data, if the format does not meet the requirements after verification, convert the format, and then perform data cleaning and normalization processing;

[0056] Data preprocessing module: Use a clustering-based outlier detection algorithm to process outliers. Determine that the neighborhood radius Eps of the core point is 0.3, and the minimum number of samples MinPts is 3. For outliers, use the method of replacing with neighboring values for correction. Then perform normalization processing on all data.

[0057] Feature engineering:

[0058] The feature engineering module extracts effective features and selects a feature subset using a feature selection algorithm based on the information gain ratio. The penalty coefficient ρ is set to 0.2. The finally selected feature subset includes the electricity load in the past 24 hours, the current temperature, humidity, and the rate of change of the electricity load in the past 12 hours, etc.

[0059] Model construction and training include a machine learning prediction model module and a model training module;

[0060] The machine learning prediction model module: constructs a prediction model including an input layer, 4 hidden layers, and an output layer. In the mixed activation function, the initial value of the weight coefficient α is 0.5, the adjustment step β is set to 0.03. In the adaptive weight adjustment mechanism, the initial value of the learning rate η is 0.005, and the activity adjustment coefficient γ is set to 0.01;

[0061] The model training module: uses the feature subset to train the model, adopts a custom loss function. The number of samples N is the number of hours in the past 3 years (about 26,280). The regularization parameter λ is determined through cross-validation. The training set is divided into 8 subsets of equal size. Each time, 7 of these subsets are selected as the training subset, and the remaining one subset is used as the validation subset. For different candidate sets of λ values (0.0001, 0.0002, 0.0003, 0.0004, 0.0005), the model is trained on each training subset, and the model performance is evaluated on the corresponding validation subset. The mean absolute error (MAE) is used as the evaluation metric, and finally the value of λ is determined to be 0.0003.

[0062] Prediction and effect monitoring:

[0063] The prediction output module inputs the data to be predicted to obtain the result, which is displayed as a bar chart through the result visualization sub-module, and a detailed analysis report is provided. The performance is monitored through RMSE, MAE, and R 2 In this embodiment, the RMSE is approximately 0.03, the MAE is approximately 0.02, and R is approximately 0.90, indicating that the model has good prediction performance.

[0064] Embodiment 3: Prediction of commodity sales

[0065] For retailers, predicting commodity sales helps to reasonably arrange inventory and procurement plans. This system is used to predict the sales volume of a certain commodity in a supermarket in the next month.

[0066] Data collection and preprocessing include a data collection module and a data preprocessing module;

[0067] Data collection module: Collect the sales data of this product in the past 2 years, including weekly sales volume, price, promotion information, etc. The time node is weekly. At the same time, through the real-time data update sub-module, connect to the data source and synchronize data every day to obtain the latest sales data. For the newly added data, after processing the format, perform cleaning and normalization processing;

[0068] Data preprocessing module: Use the outlier detection algorithm based on clustering to process outliers. The neighborhood radius Eps of the core point is 0.4, and the minimum number of samples MinPts is 4. For outliers, use the linear interpolation method to correct them, and then perform normalization processing.

[0069] Feature engineering:

[0070] Feature engineering module: Extract effective features and select a subset of features. The penalty coefficient ρ is set to 0.4. The finally selected subset of features includes the sales volume in the past 4 weeks, the current price, whether there was a promotion in the past 2 weeks, and the change rate of the sales volume in the past 8 weeks, etc.

[0071] Model construction and training include a machine learning prediction model module and a model training module;

[0072] Machine learning prediction model module: Build a prediction model including an input layer, 2 hidden layers, and an output layer. In the mixed activation function, the initial value of the weight coefficient α is 0.5, the adjustment step size β is set to 0.08. In the adaptive weight adjustment mechanism, the initial value of the learning rate η is 0.008, and the activity adjustment coefficient γ is set to 0.03;

[0073] Model training module: Use the subset of features to train the model, adopt a custom loss function, the number of samples N is the number of weeks in the past 2 years (about 104), determine the regularization parameter λ through cross-validation, divide the training set into 4 subsets of equal size, each time select 3 of them as the training subset, and the remaining one subset as the validation subset. For different candidate sets of λ values (0.0001, 0.0005, 0.0008, 0.001), train the model on each training subset and evaluate the model performance on the corresponding validation subset. Use the mean absolute error (MAE) as the evaluation index, and finally determine that the value of λ is 0.0008.

[0074] Prediction and effect monitoring:

[0075] Prediction output module: Input the data to be predicted to get the result, display it as a scatter plot through the result visualization sub-module, and provide a detailed analysis report. Monitor the performance through RMSE, MAE, and R 2 In this embodiment, RMSE is about 0.04, MAE is about 0.025, and R is about 0.88, indicating that the model has good prediction performance.

[0076] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in any regard, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. A big data prediction system based on machine learning, characterized in that, The system includes a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module, and a prediction output module; The data acquisition module is used to collect a big data sample set, which covers historical data of multiple time nodes, and the data of each time node has multiple feature variables; The data preprocessing module performs cleaning and normalization on the collected data, eliminates outliers and noise, and makes the data meet the input requirements of machine learning algorithms; The feature engineering module extracts effective features from the preprocessed data, and uses a feature selection algorithm based on information gain ratio to select a feature subset with high correlation with the prediction target; The machine learning prediction model module, the machine learning prediction model is based on an improved deep neural network architecture, including an input layer, multiple hidden layers, and an output layer, where the hidden layer adopts a mixed activation function and an adaptive weight adjustment mechanism; The hybrid activation function is \(f(x)=\alpha\times ReLu(x)+(1 - \alpha)\times Sigmoid(x)\), where \(x\) is the neuron input, \(\alpha\) is the weight coefficient, and its value is determined by a dynamic adjustment strategy based on the data distribution during the training process. The initial value is set to 0.

5. As the number of training iterations increases, it is dynamically adjusted according to the mean and variance of each batch of data. The adjustment formula is \(\alpha\) t+1 =\(\alpha\) t +\(\beta\times(\mu - \sigma)\), where \(\alpha\) t is the value of \(\alpha\) at the \(t\)-th iteration, \(\beta\) is the adjustment step size, and its value range is \([0.01, 0.1]\), \(\mu\) is the mean of the current batch of data, and \(\sigma\) is the variance of the current batch of data; The adaptive weight adjustment mechanism adopts a comprehensive evaluation method based on gradient information and neuron activity. For the weight of the $i$-th neuron in the $l$-th layer Its update formula is where, is the weight value at the $t$-th iteration, $\eta$ is the learning rate, and its value range is $[0.001, 0.1]$. is the gradient of the loss function $E$ with respect to this weight, $\gamma$ is the activity adjustment coefficient, and its value range is $[0.005, 0.05]$. is the activity of this neuron, and its calculation formula is where $T$ is the total number of iterations. is the activation value of this neuron in the $l$-th layer at the $t$-th iteration; The model training module uses the processed feature subset to train an innovative machine learning prediction model and optimizes it using a custom loss function, where the loss function is where N is the number of samples, y i is the true value of the i-th sample, is the model prediction value, λ is the regularization parameter, and its value range is [0.0001, 0.001], M is the total number of model weights, and w j is the j-th weight; The prediction output module inputs the data to be predicted into the trained model to obtain a prediction result.

2. The big data prediction system based on machine learning according to claim 1, wherein The data acquisition module also includes a real-time data update sub-module, which obtains the latest big data samples in real time through a regular connection and data synchronization mechanism with the data source to ensure the timeliness of the data. For newly added data, first perform data format verification. If the format does not meet the requirements, automatically call the format conversion tool for conversion, and then incorporate it into the data cleaning and normalization process.

3. A big data prediction system based on machine learning according to claim 1, characterized in that, The outlier processing in the data preprocessing module adopts a clustering-based outlier detection algorithm, and the specific steps are as follows: Step 1, perform clustering analysis on the data, adopt the density-based clustering algorithm DBSCAN, and divide the data points into different clusters. The neighborhood radius Eps and the minimum number of samples MinPts of the core points are adaptively determined according to the dimension and distribution characteristics of the data, and the optimal parameter combination is found through multiple experiments and evaluations; Step 2, mark the points that do not belong to any cluster as outliers, that is, outliers. For the detected outliers, according to their distance from adjacent data points and the data distribution, use linear interpolation or neighboring value replacement methods for correction.

4. A big data prediction system based on machine learning according to claim 1, characterized in that, In the feature selection algorithm based on information gain ratio in the feature engineering module, when calculating the information gain ratio, consider the correlation between features. For feature groups with high correlation, introduce a penalty factor to reduce their information gain ratio, so as to avoid selecting redundant features; The specific calculation formula is as follows: Among them, GainRatio(F) is the information gain ratio of feature F, Gain(F) is the information gain of feature F, SplitInfo(F) is the split information value of feature F, ρ is the penalty coefficient, and its value range is [0.1, 0.5]. Corr(F, F i ) is the correlation coefficient between feature F and the i-th other feature, and m is the number of features related to feature F.

5. A big data prediction system based on machine learning according to claim 1, characterized in that, The number of hidden layers in the machine learning prediction model module is automatically adjusted according to the complexity of the data and the difficulty of the prediction task, and the adjustment strategy is based on the performance evaluation index of the model on the validation set; Specifically, when the loss function of the model on the validation set no longer decreases for K consecutive iterations, a hidden layer is added, and at the same time, the number of neurons in each layer is adjusted accordingly. The initial value of the number of neurons in the newly added hidden layer is p times the number of neurons in the previous layer, where the value range of p is [0.5, 1.5], and fine-tuning is performed according to the performance during subsequent training. When overfitting occurs in the model on the validation set, a hidden layer is removed, and the neuron connection weights of the remaining hidden layers are re-initialized and optimized for training.

6. A big data prediction system based on machine learning according to claim 1, wherein, The learning rate η in the model training module adopts an adaptive adjustment strategy and is dynamically updated according to the gradient change of the model during the training process; The specific formula is as follows: where η t is the learning rate at the t-th iteration, is the gradient of the loss function at the t-th iteration, is the initial gradient of the loss function, ∈ is a very small positive number to prevent the denominator from being zero, and its value is 1e -8 , σ t is a smoothing factor based on the historical gradient information, and its calculation formula is where α is the smoothing coefficient, and its value range is [0.7, 0.9].

7. A big data prediction system based on machine learning according to claim 1, characterized in that, The regularization parameter λ used in the model training module determines its optimal value through cross-validation on the training set; The specific operations are as follows: divide the training set into M subsets of equal size. Each time, select M - 1 of these subsets as the training subset, and the remaining one subset as the validation subset. For different candidate sets of λ values {λ1, λ2, …, λ n}, train the model on each training subset and evaluate the model performance on the corresponding validation subset. Use the mean absolute error (MAE) as the evaluation metric, and select the λ value that minimizes the MAE on the validation set as the final regularization parameter for model training.

8. A big data prediction system based on machine learning according to claim 1, characterized in that, The prediction output module further includes a result visualization sub-module, which presents the prediction results to the user in an intuitive chart form, including various forms such as line charts, bar charts, and scatter plots. The user can select different chart types for viewing according to their needs; At the same time, a detailed analysis report of the prediction results is provided, including the comparison between the predicted value and the true value, error analysis, and trend analysis, to help the user better understand the prediction results and data characteristics.

9. A big data prediction system based on machine learning according to any one of claims 1-8, characterized in that The system also includes a system performance evaluation module that regularly evaluates the prediction performance of the system, and comprehensively evaluates it using the root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R 2 ) According to the evaluation results, a performance report is generated and optimization suggestions are provided. If it is found that the system performance deteriorates or anomalies occur, the model re-training or parameter adjustment mechanism is automatically triggered to ensure that the system always maintains good prediction performance.

Images