Big data prediction system based on machine learning

Abstract

The invention discloses a big data prediction system based on machine learning, and relates to the technical field of machine learning and big data, and the system comprises a data collection module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module and a prediction output module. According to the method, a specific algorithm is applied through the feature engineering module, correlation is considered, a penalty factor is introduced, related feature subsets are focused to improve precision, a hidden layer and parameters of the model module can be adaptively adjusted, for example, high precision is obtained in power load prediction, the data acquisition module updates and processes a format in real time, and data timeliness and availability are ensured. The preprocessing module processes abnormal values by adopting a clustering algorithm, corrects and normalizes parameters after determining the parameters, meets the requirements of the algorithm and improves the stability and accuracy, and the characteristics act together, so that the system is excellent in performance in a prediction task, and a solid foundation is laid for accurate prediction.

Description

Technical Field

[0001] The present invention relates to the field of machine learning and big data technology, and specifically to a big data prediction system based on machine learning. Background Art

[0002] With the widespread application of big data and machine learning technologies in various fields, traditional prediction systems have gradually exposed a series of defects.

[0003] In terms of feature engineering, most existing systems do not fully consider the correlation between features and lack effective means to avoid the selection of redundant features. This not only complicates the model structure, but also seriously restricts the accuracy of prediction.

[0004] In terms of model structure, the number of hidden layers in common prediction models is usually fixed. This fixed structure cannot be adaptively adjusted according to the complexity of the data and the difficulty of the prediction task, making it difficult to adapt to different data characteristics, which has an adverse impact on the prediction effect.

[0005] In the data collection module, many systems have obvious deficiencies. On the one hand, there is a lack of effective real-time data update mechanism, and the latest big data samples cannot be obtained in time, resulting in poor data timeliness. On the other hand, for the newly added data, either the format is not verified or it cannot be properly processed when the format does not meet the requirements, which directly affects the integrity and availability of the data, and thus hinders the subsequent prediction work.

[0006] There are also problems in the data preprocessing stage. In terms of outlier processing, most systems do not determine appropriate parameters based on the characteristics of the data, and the processing methods are not scientific and reasonable. There are also deficiencies in data normalization processing, making it difficult for the data to meet the strict requirements of machine learning algorithms, ultimately leading to a decrease in the stability and accuracy of the model. Summary of the invention

[0007] The purpose of the present invention is to make up for the shortcomings of the prior art and provide a big data prediction system based on machine learning. It can use a specific algorithm through the feature engineering module, consider the correlation and introduce a penalty factor, focus on the relevant feature subset to improve the accuracy, and the hidden layer and parameters of the model module can be adjusted adaptively, such as obtaining high accuracy in power load forecasting. The data acquisition module updates and processes the format in real time to ensure the timeliness and availability of the data. The preprocessing module uses a clustering algorithm to process abnormal values, and corrects and normalizes the parameters after determining them to meet the algorithm requirements and improve stability and accuracy. These characteristics work together to enable the system to perform well in the prediction task and lay a solid foundation for accurate prediction.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: a big data prediction system based on machine learning, the system comprising a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module and a prediction output module;

[0009] The data collection module is used to collect a large data sample set, which covers historical data of multiple time nodes, and the data of each time node has multiple characteristic variables;

[0010] The data preprocessing module cleans and normalizes the collected data to eliminate outliers and noise, so that the data meets the input requirements of the machine learning algorithm;

[0011] The feature engineering module extracts effective features from the preprocessed data and selects a feature subset with high relevance to the prediction target using a feature selection algorithm based on information gain ratio;

[0012] The machine learning prediction model module, the machine learning prediction model is based on an improved deep neural network architecture, comprising an input layer, multiple hidden layers and an output layer, wherein the hidden layer adopts a hybrid activation function and an adaptive weight adjustment mechanism;

[0013] The hybrid activation function is f(x)=α×ReLU(x)+(1-α)×Sigmoid(x), where x is the neuron input and α is the weight coefficient, whose value is determined by a dynamic adjustment strategy based on data distribution during training. The initial value is set to 0.5. As the number of training iterations increases, it is dynamically adjusted according to the mean and variance of each batch of data. The adjustment formula is α t+1 =α t +β×μ-σ), where α t is the α value at the t-th iteration, β is the adjustment step size, the value range is [0.01, 0.1], μ is the mean of the current batch data, and σ is the variance of the current batch data;

[0014] The adaptive weight adjustment mechanism adopts a comprehensive evaluation method based on gradient information and neuron activity. Its update formula is in is the weight value at the tth iteration, η is the learning rate, and its value range is [0.001, 0.1], is the gradient of the loss function E with respect to the weight, γ is the activity adjustment coefficient, and its value range is [0.005, 0.05]. is the activity of the neuron, and the calculation formula is Where T is the total number of iterations, is the activation value of the neuron in the lth layer at the tth iteration;

[0015] The model training module uses the processed feature subset to train the innovative machine learning prediction model and optimizes it using a custom loss function. The loss function is Where N is the number of samples, y i is the true value of the i-th sample, is the model prediction value, λ is the regularization parameter, the value range is [0.0001, 0.001], M is the total number of model weights, w j is the jth weight;

[0016] The prediction output module inputs the data to be predicted into the trained model to obtain the prediction result.

[0017] Furthermore, the data acquisition module also includes a real-time data update submodule, which obtains the latest big data samples in real time through regular connection with the data source and data synchronization mechanism to ensure the timeliness of the data. For the newly added data, the data format is first verified. If the format does not meet the requirements, the format conversion tool is automatically called for conversion, and then it is included in the data cleaning and normalization processing flow.

[0018] Furthermore, the outlier processing in the data preprocessing module adopts a clustering-based outlier detection algorithm, and the specific steps are as follows:

[0019] Step 1: Perform cluster analysis on the data and use the density-based clustering algorithm DBSCAN to divide the data points into different clusters. The neighborhood radius Eps of the core point and the minimum number of samples MinPts are adaptively determined according to the dimension and distribution characteristics of the data. The optimal parameter combination is found through multiple experiments and evaluations.

[0020] Step 2: Mark the points that do not belong to any cluster as outliers, i.e., outliers. For the detected outliers, they are corrected by linear interpolation or neighboring value replacement according to their distance from adjacent data points and data distribution.

[0021] Furthermore, the feature selection algorithm based on information gain ratio in the feature engineering module considers the correlation between features when calculating the information gain ratio, and for feature groups with high correlation, reduces their information gain ratio by introducing a penalty factor, thereby avoiding the selection of redundant features;

[0022] The specific calculation formula is: Where GainRatio(F) is the information gain ratio of feature F, Gain(F) is the information gain of feature F, SplitInfo(F) is the split information value of feature F, ρ is the penalty coefficient, and the value range is [0.1, 0.5]. Corr(F,F i ) is the correlation coefficient between feature F and the i-th other feature, and n is the number of features related to feature F.

[0023] Furthermore, the number of hidden layers in the machine learning prediction model module is automatically adjusted according to the complexity of the data and the difficulty of the prediction task, and the adjustment strategy is based on the performance evaluation index of the model on the validation set;

[0024] Specifically, when the loss function of the model on the validation set no longer decreases after K consecutive iterations, a hidden layer is added, and the number of neurons in each layer is adjusted accordingly. The initial value of the number of neurons in the newly added hidden layer is p times the number of neurons in the previous layer, and the value range of p is [0.5, 1.5]. It is fine-tuned according to the performance in the subsequent training process. When the model overfits on the validation set, one hidden layer is reduced, and the neuron connection weights of the remaining hidden layers are reinitialized and optimized.

[0025] Furthermore, the learning rate η in the model training module adopts an adaptive adjustment strategy and is dynamically updated according to the gradient changes of the model during the training process;

[0026] The specific formula is: where η t is the learning rate at the tth iteration, is the gradient of the loss function at the tth iteration, is the initial loss function gradient, ∈ is a very small positive number to prevent the denominator from being zero, and the value is 1e -8 , σ t is a smoothing factor based on gradient history information, and the calculation formula is Where α is the smoothing coefficient, and its value range is [0.7, 0.9].

[0027] Furthermore, the regularization parameter λ used in the model training module is determined to have its optimal value by cross-validation on the training set;

[0028] The specific operation is as follows: divide the training set into M subsets of equal size, select M-1 subsets as training subsets each time, and the remaining subset as validation subset. For different λ value candidate sets {λ1,λ2,…,λ n}, train the model on each training subset and evaluate the model performance on the corresponding validation subset. The mean absolute error (MAE) is used as the evaluation indicator, and the λ value that minimizes the MAE on the validation set is selected as the final regularization parameter for model training.

[0029] Furthermore, the prediction output module also includes a result visualization submodule, which displays the prediction results to the user in the form of intuitive charts, including line charts, bar charts and scatter charts. Users can select different chart types for viewing according to their needs;

[0030] At the same time, a detailed analysis report of the prediction results is provided, including comparison between the predicted value and the actual value, error analysis and trend analysis, to help users better understand the prediction results and data characteristics.

[0031] Furthermore, the system also includes a system performance evaluation module, which regularly evaluates the prediction performance of the system using root mean square error (RMSE), mean absolute error (MAE) and coefficient of determination (R 2 ) to conduct a comprehensive evaluation;

[0032] Based on the evaluation results, a performance report is generated and optimization suggestions are provided. If system performance deteriorates or anomalies occur, the model retraining or parameter adjustment mechanism is automatically triggered to ensure that the system always maintains good predictive performance.

[0033] Compared with the existing technology, this big data prediction system based on machine learning has the following beneficial effects:

[0034] 1. The feature engineering module of the present invention uses a feature selection algorithm based on information gain ratio, and considers the correlation between features. By introducing a penalty factor, redundant features are avoided, which enables the model to focus on a feature subset that is highly relevant to the prediction target, thereby improving the prediction accuracy of the model. The number of hidden layers of the machine learning prediction model module can be automatically adjusted according to the complexity of the data and the difficulty of the prediction task. At the same time, the weight coefficient α of the hybrid activation function and the learning rate η, activity adjustment coefficient γ and other parameters in the adaptive weight adjustment mechanism can be dynamically adjusted. In the power load prediction embodiment, this adaptive adjustment mechanism enables the model to better adapt to the characteristics of the power load data, and ultimately obtain a higher prediction accuracy.

[0035] 2. The real-time data update submodule of the data acquisition module of the present invention can obtain the latest big data samples in real time through regular connection and data synchronization mechanism to ensure the timeliness of the data. For the newly added data, the data format is first verified. If the format does not meet the requirements, the format conversion tool is automatically called for conversion, and the data format is effectively processed to ensure the integrity and availability of the data, laying the foundation for subsequent accurate predictions. The data preprocessing module uses a clustering-based outlier detection algorithm to process outliers, and determines appropriate parameters according to the characteristics of the data, and then uses linear interpolation or neighboring value replacement and other methods to make corrections, and then normalizes all data. In this way, the outliers are effectively processed, so that the data meets the input requirements of the machine learning algorithm and improves the stability and accuracy of the model.

[0036] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 The figure is a flowchart of a big data prediction system based on machine learning. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1: Stock market price prediction

[0041] In the stock market, predicting the trend of stock prices is crucial for investors. This application uses a big data prediction system based on machine learning to predict the price of a certain stock in the future.

[0042] Data collection and preprocessing include data collection module and data preprocessing module;

[0043] Data collection module: collects historical trading data of the stock from the stock trading database, including opening price, closing price, highest price, lowest price, trading volume and other characteristic variables, covering the past 5 years, multiple time nodes, and each trading day as a time node. At the same time, through the real-time data update submodule, connect and synchronize data with the data source every 15 minutes to obtain the latest trading data. For the newly added data, first verify whether its format meets the requirements. If not, call the format conversion tool to convert it into a format that the system can handle, and then include it in the data cleaning and normalization processing process.

[0044] Data preprocessing module: DBSCAN is used to process outliers. According to the dimension and distribution characteristics of the data, the neighborhood radius Eps of the core point is determined to be 0.5, and the minimum number of samples MinPts is 5. Points that do not belong to any cluster are marked as outliers. For these outliers, linear interpolation is used to correct them according to their distance from adjacent data points and data distribution. All data are then normalized to meet the input requirements of the machine learning algorithm.

[0045] Feature Engineering:

[0046] The feature engineering module extracts effective features from the preprocessed data and uses the feature selection algorithm based on the information gain ratio to select feature subsets with high correlation with the prediction target. When calculating the information gain ratio, the correlation between features is considered and the penalty coefficient ρ is set to 0.3. The final selected feature subset includes the closing price and trading volume of the past 10 trading days and the price volatility of the past 5 trading days.

[0047] Model building and training include machine learning prediction model module and model training module;

[0048] Machine learning prediction model module: Build a prediction model based on an improved deep neural network architecture, which includes an input layer, three hidden layers and an output layer. The hidden layer uses a hybrid activation function and an adaptive weight adjustment mechanism. In the hybrid activation function, the initial value of the weight coefficient α is 0.5, and the adjustment step size β is set to 0.05. In the adaptive weight adjustment mechanism, the initial value of the learning rate η is 0.01, and the activity adjustment coefficient γ is set to 0.02;

[0049] Model training module: The model is trained using the processed feature subsets and optimized using a custom loss function. The number of samples N in the loss function is the number of trading days in the past five years (about 1,250). The regularization parameter λ is determined through cross-validation. The training set is divided into five subsets of equal size, and four subsets are selected as training subsets each time, and the remaining subset is used as a validation subset. For different candidate sets of λ values ​​(0.0001, 0.0003, 0.0005, 0.0007, and 0.0009), the model is trained on each training subset and the model performance is evaluated on the corresponding validation subset. The mean absolute error (MAE) is used as the evaluation indicator, and the value of λ is finally determined to be 0.0005.

[0050] Prediction and effect monitoring:

[0051] The prediction output module inputs the stock data to be predicted into the trained model to obtain the prediction results. The result visualization submodule displays the prediction results to the user in the form of a line graph and provides a detailed analysis report of the prediction results, including the comparison between the predicted value and the true value, error analysis, trend analysis, etc. The root mean square error (RMSE), mean absolute error (MAE) and determination coefficient (R 2 ) The prediction performance is monitored. In this embodiment, after a period of monitoring, the RMSE is about 0.05, the MAE is about 0.03, and the MAE is about 0.85, indicating that the model has good prediction performance.

[0052] Example 2: Power load forecasting

[0053] In the power system, accurate prediction of power load is crucial for rationally arranging power generation plans and ensuring stable power supply. This system is used to predict the power load of a certain area in the next week.

[0054] Data acquisition and preprocessing include data acquisition module and data preprocessing module;

[0055] Data collection module: collects the power load data and related meteorological data (such as temperature, humidity, wind speed, etc.) of the region in the past three years, with a time node of every hour. At the same time, through the real-time data update submodule, it connects with the data source and synchronizes data every hour to obtain the latest power load and meteorological data. For the newly added data, after verifying the format, if it does not meet the requirements, it will convert the format, and then perform data cleaning and normalization processing;

[0056] Data preprocessing module: The clustering-based outlier detection algorithm is used to process outliers, and the neighborhood radius Eps of the core point is determined to be 0.3, and the minimum number of samples MinPts is 3. For outliers, the neighboring value replacement method is used to correct them. Then all data are normalized.

[0057] Feature Engineering:

[0058] The feature engineering module extracts effective features and uses a feature selection algorithm based on information gain ratio to select feature subsets. The penalty coefficient ρ is set to 0.2. The final selected feature subset includes the power load in the past 24 hours, current temperature, humidity, and the power load change rate in the past 12 hours.

[0059] Model building and training include machine learning prediction model module and model training module;

[0060] Machine learning prediction model module: Build a prediction model consisting of an input layer, four hidden layers, and an output layer. In the hybrid activation function, the initial value of the weight coefficient α is 0.5, the adjustment step size β is set to 0.03, and in the adaptive weight adjustment mechanism, the initial value of the learning rate η is 0.005, and the activity adjustment coefficient γ is set to 0.01.

[0061] Model training module: The model is trained using feature subsets, with a custom loss function. The sample size N is the number of hours in the past three years (about 26,280). The regularization parameter λ is determined by cross-validation, and the training set is divided into 8 subsets of equal size. Seven subsets are selected as training subsets each time, and the remaining subset is used as a validation subset. For different candidate sets of λ values ​​(0.0001, 0.0002, 0.0003, 0.0004, and 0.0005), the model is trained on each training subset and the model performance is evaluated on the corresponding validation subset. The mean absolute error (MAE) is used as the evaluation indicator, and the final λ value is determined to be 0.0003.

[0062] Prediction and effect monitoring:

[0063] The prediction output module inputs the data to be predicted to obtain the results, which are displayed in a bar chart through the result visualization submodule and provide a detailed analysis report, using RMSE, MAE and R 2 Monitoring performance, in this embodiment, the RMSE is about 0.03, the MAE is about 0.02, and the P is about 0.90, which shows that the model prediction performance is good.

[0064] Example 3: Product Sales Forecast

[0065] For retailers, forecasting product sales helps to arrange inventory and purchase plans reasonably. Use this system to predict the sales volume of a certain product in a supermarket in the next month.

[0066] Data acquisition and preprocessing include data acquisition module and data preprocessing module;

[0067] Data collection module: collects the sales data of the product in the past two years, including weekly sales volume, price, promotional information, etc., with a time node of every week. At the same time, through the real-time data update submodule, it connects with the data source and synchronizes data every day to obtain the latest sales data. For the newly added data, it processes the format and then cleans and normalizes it;

[0068] Data preprocessing module: A clustering-based outlier detection algorithm is used to process outliers. The neighborhood radius Eps of the core point is 0.4, and the minimum number of samples MinPts is 4. For outliers, linear interpolation method is used to correct them, and then normalization is performed.

[0069] Feature Engineering:

[0070] The feature engineering module extracts effective features and selects feature subsets. The penalty coefficient ρ is set to 0.4. The final selected feature subset includes sales in the past 4 weeks, current price, whether there have been promotional activities in the past 2 weeks, and the rate of change of sales in the past 8 weeks.

[0071] Model building and training include machine learning prediction model module and model training module;

[0072] Machine learning prediction model module: Build a prediction model consisting of an input layer, two hidden layers, and an output layer. In the hybrid activation function, the initial value of the weight coefficient α is 0.5, the adjustment step size β is set to 0.08, and in the adaptive weight adjustment mechanism, the initial value of the learning rate η is 0.008, and the activity adjustment coefficient γ is set to 0.03;

[0073] Model training module: The model is trained using feature subsets, with a custom loss function. The sample size N is the number of weeks in the past two years (about 104). The regularization parameter λ is determined by cross-validation, and the training set is divided into four subsets of equal size. Three subsets are selected as training subsets each time, and the remaining subset is used as a validation subset. For different candidate sets of λ values ​​(0.0001, 0.0005, 0.0008, 0.001), the model is trained on each training subset and the model performance is evaluated on the corresponding validation subset. The mean absolute error (MAE) is used as the evaluation indicator, and the final λ value is determined to be 0.0008.

[0074] Prediction and effect monitoring:

[0075] The prediction output module inputs the data to be predicted to obtain the result, which is displayed in a scatter plot through the result visualization submodule and provides a detailed analysis report. 2 Monitoring performance, in this example, the RMSE is about 0.04, the MAE is about 0.025, and the P is about 0.88, indicating that the model has good predictive performance.

[0076] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A big data prediction system based on machine learning, characterized in that: The system includes a data acquisition module, a data preprocessing module, a feature engineering module, a machine learning prediction model module, a model training module and a prediction output module; The data collection module is used to collect a large data sample set, which covers historical data of multiple time nodes, and the data of each time node has multiple characteristic variables; The data preprocessing module cleans and normalizes the collected data to eliminate outliers and noise, so that the data meets the input requirements of the machine learning algorithm; The feature engineering module extracts effective features from the preprocessed data and selects a feature subset with high relevance to the prediction target using a feature selection algorithm based on information gain ratio; The machine learning prediction model module, the machine learning prediction model is based on an improved deep neural network architecture, comprising an input layer, multiple hidden layers and an output layer, wherein the hidden layer adopts a hybrid activation function and an adaptive weight adjustment mechanism; The hybrid activation function is f(x)=α×ReLu(x)+(1-α)×Sigmoid(x), where x is the neuron input and α is the weight coefficient, whose value is determined by a dynamic adjustment strategy based on data distribution during training. The initial value is set to 0.

5. As the number of training iterations increases, it is dynamically adjusted according to the mean and variance of each batch of data. The adjustment formula is α t+1 =α t +β×(μ-σ), where α t is the α value at the t-th iteration, β is the adjustment step size, the value range is [0.01, 0.1], μ is the mean of the current batch data, and σ is the variance of the current batch data; The adaptive weight adjustment mechanism adopts a comprehensive evaluation method based on gradient information and neuron activity. Its update formula is in, is the weight value at the tth iteration, η is the learning rate, and its value range is [0.001, 0.1], is the gradient of the loss function E with respect to the weight, γ is the activity adjustment coefficient, and its value range is [0.005, 0.05]. is the activity of the neuron, and the calculation formula is Where T is the total number of iterations, is the activation value of the neuron in the lth layer at the tth iteration; The model training module uses the processed feature subset to train the innovative machine learning prediction model and optimizes it using a custom loss function. The loss function is Where N is the number of samples, y i is the true value of the i-th sample, is the model prediction value, λ is the regularization parameter, the value range is [0.0001, 0.001], M is the total number of model weights, w j is the jth weight; The prediction output module inputs the data to be predicted into the trained model to obtain the prediction result.

2. The big data prediction system based on machine learning according to claim 1, characterized in that: The data acquisition module also includes a real-time data update submodule, which obtains the latest big data samples in real time through regular connection with the data source and data synchronization mechanism to ensure the timeliness of the data. For the newly added data, the data format is first verified. If the format does not meet the requirements, the format conversion tool is automatically called for conversion, and then it is included in the data cleaning and normalization processing flow.

3. The big data prediction system based on machine learning according to claim 1, characterized in that: The outlier processing in the data preprocessing module adopts a clustering-based outlier detection algorithm, and the specific steps are as follows: Step 1: Perform cluster analysis on the data and use the density-based clustering algorithm DBSCAN to divide the data points into different clusters. The neighborhood radius Eps of the core point and the minimum number of samples MinPts are adaptively determined according to the dimension and distribution characteristics of the data. The optimal parameter combination is found through multiple experiments and evaluations. Step 2: Mark the points that do not belong to any cluster as outliers, i.e., outliers. For the detected outliers, they are corrected by linear interpolation or neighboring value replacement according to their distance from adjacent data points and data distribution.

4. The big data prediction system based on machine learning according to claim 1, characterized in that: The feature selection algorithm based on information gain ratio in the feature engineering module considers the correlation between features when calculating the information gain ratio. For feature groups with high correlation, a penalty factor is introduced to reduce their information gain ratio, thereby avoiding the selection of redundant features. The specific calculation formula is: Where GainRatio(F) is the information gain ratio of feature F, Gain(F) is the information gain of feature F, SplitInfo(F) is the split information value of feature F, ρ is the penalty coefficient, and the value range is [0.1, 0.5]. Corr(F,F i ) is the correlation coefficient between feature F and the i-th other feature, and m is the number of features related to feature F.

5. The big data prediction system based on machine learning according to claim 1, characterized in that: The number of hidden layers in the machine learning prediction model module is automatically adjusted according to the complexity of the data and the difficulty of the prediction task, and the adjustment strategy is based on the performance evaluation index of the model on the validation set; Specifically, when the loss function of the model on the validation set no longer decreases after K consecutive iterations, a hidden layer is added, and the number of neurons in each layer is adjusted accordingly. The initial value of the number of neurons in the newly added hidden layer is p times the number of neurons in the previous layer, and the value range of p is [0.5, 1.5]. It is fine-tuned according to the performance in the subsequent training process. When the model overfits on the validation set, one hidden layer is reduced, and the neuron connection weights of the remaining hidden layers are reinitialized and optimized.

6. The big data prediction system based on machine learning according to claim 1, characterized in that: The learning rate η in the model training module adopts an adaptive adjustment strategy and is dynamically updated according to the gradient changes of the model during the training process; The specific formula is: where η t is the learning rate at the tth iteration, is the gradient of the loss function at the tth iteration, is the initial loss function gradient, ∈ is a very small positive number to prevent the denominator from being zero, and the value is 1e -8 , σ t is a smoothing factor based on the gradient history information, and its calculation formula is Where α is the smoothing coefficient, and its value range is [0.7, 0.9].

7. The big data prediction system based on machine learning according to claim 1, characterized in that: The regularization parameter λ used in the model training module is determined to have its optimal value by cross-validation on the training set; The specific operation is as follows: divide the training set into M subsets of equal size, select M-1 subsets as training subsets each time, and the remaining subset as validation subset. For different λ value candidate sets {λ1,λ2,…,λ n }, train the model on each training subset and evaluate the model performance on the corresponding validation subset. The mean absolute error (MAE) is used as the evaluation indicator, and the λ value that minimizes the MAE on the validation set is selected as the final regularization parameter for model training.

8. The big data prediction system based on machine learning according to claim 1, characterized in that: The prediction output module also includes a result visualization submodule, which displays the prediction results to the user in the form of intuitive charts, including line charts, bar charts, scatter charts and other forms. Users can select different chart types for viewing according to their needs; At the same time, a detailed analysis report of the prediction results is provided, including comparison between the predicted value and the actual value, error analysis and trend analysis, to help users better understand the prediction results and data characteristics.

9. A big data prediction system based on machine learning according to any one of claims 1 to 8, characterized in that: The system also includes a system performance evaluation module, which regularly evaluates the prediction performance of the system using root mean square error (RMSE), mean absolute error (MAE) and coefficient of determination (R 2 ) to conduct a comprehensive evaluation; Based on the evaluation results, a performance report is generated and optimization suggestions are provided. If system performance deteriorates or anomalies occur, the model retraining or parameter adjustment mechanism is automatically triggered to ensure that the system always maintains good predictive performance.

Images