High-dimensional data prediction processing method based on big data technology
By using data crawling technology, feature selection and gradient descent method in financial data prediction, linear equation systems are constructed to process high-dimensional financial data, solving the problems of overfitting, long calculation time and insufficient prediction accuracy in the existing technology, and achieving more efficient and accurate financial data prediction.
Patent Information
- Application Number
- CN202411600968.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2025-03-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing prediction processing technology methods are prone to overfitting when processing high-dimensional data, the calculation time is too long, and the real-time decision-making needs cannot be met. Especially in financial data prediction, the calculation process under high-dimensional data is unstable and the prediction accuracy is insufficient.
Data is collected from financial data sources through data crawling technology, and denoising, outliers and missing values are processed. A matrix pair of financial data in the form of (Xi, Yi) is constructed, feature selection and sub-matrix construction are carried out, linear equation systems are established, and the weight matrix is solved through gradient descent method to realize the prediction processing of high-dimensional data.
It reduces the complexity of the model, improves the computing efficiency and prediction accuracy, enhances the real-time and adaptability of the model, and can more accurately predict the trends of the financial market.
Smart Images
Figure CN119671726A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a high-dimensional data prediction and processing method based on big data technology. Background Art
[0002] With the vigorous development of big data technology, data predictive analysis has penetrated into all areas of social economy and has become an indispensable force in promoting decision-making optimization and business innovation. Big data processing technology can deeply explore the inherent regularity and trend characteristics of data by efficiently and accurately collecting, storing, processing and analyzing massive data resources. These characteristics not only reveal the complex relationship between data, but also provide us with valuable clues for insight into the future and predicting trends. With the help of big data processing technology, we can conduct in-depth mining and detailed analysis of various types of data, so as to build accurate prediction models. These models can scientifically predict and analyze possible future events or trends based on historical data. Whether it is market trends, consumer behavior, disease transmission, or natural disasters, big data predictive analysis can provide us with powerful decision-making support;
[0003] However, the existing technical methods for prediction processing mainly rely on the direct calculation and solution of matrices, which can easily lead to overfitting. The model overfits the training data and lacks generalization ability. The calculation time is too long. Especially when processing big data, it cannot meet the real-time decision-making needs. It is also limited by computational complexity and data dimension. The calculation process is unstable and the prediction accuracy is insufficient under high-dimensional data, making it difficult to achieve real-time calculation. Summary of the invention
[0004] The purpose of the present invention is to provide a high-dimensional data prediction and processing method based on big data technology.
[0005] In order to solve the problems raised in the above background technology, the present invention provides the following technical solution: a high-dimensional data prediction processing method based on big data technology, the prediction method specifically comprises the following steps:
[0006] Step 1: Use data crawler technology to collect data from big data sources in the financial field, including trading data from stock exchanges, financial news media platforms, discussions related to financial markets on social media, internal databases of financial institutions, etc., and ensure the quality and reliability of the data by removing noise, outliers and missing values;
[0007] Step 2: Construct the overall data into a matrix pair (X i , Y i ) in the form of i is a matrix of financial data with a dimension of m×n, where m can represent the number of features of financial data, such as technical indicators and macro indicators of different stocks, and n can represent the number of samples.i is the corresponding target variable vector and m>n;
[0008] Step 3: For each X in the data i Perform feature selection, including but not limited to selecting moving average, relative strength index, and GDP growth rate, and then define X i Become a sub-matrix The submatrix is a rank n matrix;
[0009] Step 4: Construct a new matrix and name it Xa, in the form of (Xa, I(mn)(mn)), where I(mn)(mn)
[0010] is an identity matrix, and set
[0011] Step 5: Establish a linear system of equations based on the constructed matrix and subarrays in, represents,the weight parameter of the model, which determines the influence of the,specification financial characteristics on the target variable;
[0012] Step 6: Solve the linear equations by gradient descent method and convert the weight matrix obtained by the solution into The final weight matrix is obtained by averaging
[0013] Step 7: Processing of new data For a new input matrix X * , use the method in step 3 to construct its submatrix Ensure that the new data has the same structure and properties as the training data and predict the result Y * =X * W * , where W * is the final weight matrix;
[0014] Step 8: Evaluate the prediction results. The specific method is to use the mean square error or mean absolute error to measure the gap between the predicted value and the true value. If the evaluation result shows that the prediction accuracy of the model is not high, you can adjust the feature selection method or add new features.
[0015] As a further solution of the present invention: the data collected by the financial data collection mentioned in step 1 includes but is not limited to stock price data (opening price, closing price, highest price, lowest price and trading volume), prices of different types of bonds and foreign exchange rate data.
[0016] As a further solution of the present invention: the number of features m in the step 2 will be much larger than the number of samples n. This is because financial data often has high-dimensional characteristics. For this reason, m can represent a large number of financial feature variables, and n can represent different time points or transaction instances. Therefore, when analyzing stock market data, a large number of technical indicators and macroeconomic variables will appear as features, and the number of samples is daily trading data or observations within a specific time period.
[0017] As a further solution of the present invention: the feature selection mentioned in the step three is intended to select the most informative features from the original data set to improve the performance of the model and the accuracy of prediction. For example, features such as moving average, relative strength index and GDP growth rate can be selected. The selection of these features is based on their influence and relevance to the financial market or specific stocks. In addition to these features, other features related to the financial market can also be considered, such as trading volume, price-earnings ratio, price-to-book ratio, financial ratios, etc. The construction of the submatrix helps to reduce the complexity of the model, and feature selection and submatrix construction is an iterative and optimization process. In subsequent steps, if the evaluation results show that the prediction accuracy of the model is not high, the feature selection method can be adjusted or new features can be added to improve the model.
[0018] As a further solution of the present invention: in the step 4, a new matrix Xa is constructed based on the sub-matrix after the features. In the prediction result evaluation in step 8, if the prediction accuracy of the model is not high, the new matrix construction method in step 4 can be adjusted, including increasing or decreasing the dimension of the unit matrix, and changing the feature selection strategy to improve the prediction accuracy and robustness of the model.
[0019] As a further solution of the present invention: In the step five, a linear equation group is constructed based on the constructed new matrix Xa and the corresponding target variable vector Yi, and this equation group is the core of the model, which expresses the relationship between the features and the target variables in mathematical form. The linear equation group is constructed to solve the weight parameters, that is, to find a set of optimal weight values so that the prediction error of the model on the training data is minimized.
[0020] As a further solution of the present invention: In step 6, the gradient descent method is used to solve the linear equations. This is because the gradient descent method performs well in processing large-scale data sets and complex models. By continuously iterating and adjusting weight parameters, the gradient descent method can gradually approach the optimal solution. The specific calculation steps are as follows:
[0021] Assume the loss function is in‖*‖ 2 Represents the square of the two norm. In the gradient descent method, the formula for updating the weight vector Wi at each iteration is Where α is the learning rate, The loss function is The gradient at and initialize the weight vector Then repeat the above calculation steps until the loss function converges or reaches the maximum number of iterations to obtain multiple weight matrices W i , and finally get the weight matrix
[0022] As a further solution of the present invention: Step 7 takes over the work of data preparation, feature selection, solving a system of linear equations and averaging the weight matrix in the previous steps, and at the same time provides a basis for the result evaluation of Step 8. If the prediction result is inaccurate, the feature selection method can be adjusted or new features can be added through the feedback mechanism to further optimize the model and improve its prediction accuracy.
[0023] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are:
[0024] 1. The present invention can select the most informative features from a large number of financial features through feature selection and submatrix construction, reduce the complexity of the model, and improve computational efficiency. At the same time, by constructing a new matrix and a linear equation group, the relationship between the financial features and the target variable is expressed in mathematical form, so that the model can more accurately predict the trend of the financial market. The gradient descent method solves the linear equation group and the weight matrix averaging process further improves the accuracy and stability of the model. When processing large-scale financial data, the above method can be used to simplify the complex market prediction model into the solution of a linear equation group, further improving the real-time and accuracy of the prediction;
[0025] 2. The present invention can construct a submatrix using the same method as the training data by processing new data, ensuring that the new data has the same structure and properties as the training data, so that the model can continuously adapt to the ever-changing financial market environment. When the prediction results are inaccurate, the model can be improved by adjusting the feature selection method or adding new features. This feedback mechanism enables the model to continuously learn and optimize, thereby improving the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 Schematic diagram of the method steps in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0028] Example 1, please refer to the attached Figure 1 The present invention provides a technical solution: the prediction method specifically comprises the following steps:
[0029] Step 1: Use data crawler technology to collect data from big data sources in the financial field, including trading data from stock exchanges, financial news media platforms, discussions related to financial markets on social media, internal databases of financial institutions, etc., and ensure the quality and reliability of the data by removing noise, outliers and missing values;
[0030] Step 2: Construct the overall data into a matrix pair X i , Y i ) in the form of i is a matrix of financial data with a dimension of m×n, where m can represent the number of features of financial data, such as technical indicators and macro indicators of different stocks, and n can represent the number of samples. i is the corresponding target variable vector and m>n;
[0031] Step 3: For each X in the data i Perform feature selection, including but not limited to selecting moving average, relative strength index, and GDP growth rate, and then define X i Become a sub-matrix The submatrix is a rank n matrix;
[0032] Step 4: Construct a new matrix and name it Xa, in the form of (Xa, I(mn)(mn)), where I(mn)(mn)
[0033] is an identity matrix, and set
[0034] Step 5: Establish a linear system of equations based on the constructed matrix and subarrays in, represents,the weight parameter of the model, which determines the influence of the,specification financial characteristics on the target variable;
[0035] Step 6: Solve the linear equations by gradient descent method and convert the weight matrix obtained by the solution into The final weight matrix is obtained by averaging
[0036] Step 7: Processing of new data For a new input matrix X * , use the method in step 3 to construct its submatrix Ensure that the new data has the same structure and properties as the training data and predict the result Y * =X * W* , where W * is the final weight matrix;
[0037] Step 8: Evaluate the prediction results. The specific method is to use the mean square error or mean absolute error to measure the gap between the predicted value and the true value. If the evaluation result shows that the prediction accuracy of the model is not high, you can adjust the feature selection method or add new features.
[0038] Please refer to the attached Figure 1 The data collected by the financial data collection mentioned in step one include but are not limited to the stock price data opening price, closing price, highest price, lowest price and trading volume, the prices of different types of bonds and foreign exchange rate data. The number of features m in step two will be much larger than the number of samples n. This is because financial data often has high-dimensional characteristics. For this reason, m can represent a large number of financial feature variables, and n can represent different time points or transaction instances. Therefore, when analyzing stock market data, a large number of technical indicators and macroeconomic variables will appear as features, and the number of samples is daily trading data or observations within a specific time period. The feature selection mentioned in step three aims to select the most informative features from the original data set to improve the performance of the model and the accuracy of the prediction. For example, the moving average, relative strength index and GDP growth can be selected. The selection of these features is based on their influence and relevance on the financial market or specific stocks. In addition to these features, other features related to the financial market can also be considered, such as trading volume, price-earnings ratio, price-to-book ratio, financial ratio, etc. The construction of the sub-matrix helps to reduce the complexity of the model, and feature selection and sub-matrix construction is an iterative and optimization process. In subsequent steps, if the evaluation results show that the prediction accuracy of the model is not high, the feature selection method can be adjusted or new features can be added to improve the model. In step 4, a new matrix Xa is constructed based on the sub-matrix after the features. In the prediction result evaluation of step 8, the prediction accuracy of the model is not high, then the new matrix construction method in step 4 can be adjusted, including increasing or decreasing the dimension of the unit matrix, and changing the feature selection strategy to improve the prediction accuracy and robustness of the model.
[0039] In this embodiment, data is collected from multiple financial data sources through data crawler technology to ensure the richness and diversity of the data. This enables the model to fully consider various factors that affect the financial market. For example, the trading data of the stock exchange reflects the actual trading situation of the market, and discussions on financial news media platforms and social media provide information on market sentiment and expectations. The internal database of the financial institution contains professional financial analysis data. During the data processing process, noise, outliers and missing values are removed to ensure the quality and reliability of the data, laying a solid foundation for subsequent accurate predictions.
[0040] When in use, through feature selection and sub-matrix construction, the most informative features can be selected from a large number of financial features, reducing the complexity of the model and improving computational efficiency. At the same time, by constructing new matrices and linear equations, the relationship between financial features and target variables is expressed in mathematical form, enabling the model to more accurately predict financial market trends. The gradient descent method for solving linear equations and the weight matrix averaging process further improves the accuracy and stability of the model. When processing large-scale financial data, the above method can be used to simplify complex market forecasting models into solutions to linear equations, further improving the real-time and accuracy of forecasts.
[0041] Example 2, please refer to the attached Figure 1 The present invention provides a technical solution: the prediction method specifically comprises the following steps:
[0042] Step 1: Use data crawler technology to collect data from big data sources in the financial field, including trading data from stock exchanges, financial news media platforms, discussions related to financial markets on social media, internal databases of financial institutions, etc., and ensure the quality and reliability of the data by removing noise, outliers and missing values;
[0043] Step 2: Construct the overall data into a matrix pair X i , Y i ) in the form of i is a matrix of financial data with a dimension of m×n, where m can represent the number of features of financial data, such as technical indicators and macro indicators of different stocks, and n can represent the number of samples. i is the corresponding target variable vector and m>n;
[0044] Step 3: For each X in the data i Perform feature selection, including but not limited to selecting moving average, relative strength index, and GDP growth rate, and then define X i Become a sub-matrix The submatrix is a rank n matrix;
[0045] Step 4: Construct a new matrix and name it Xa, in the form of (Xa, I(mn)(mn)), where I(mn)(mn)
[0046] is an identity matrix, and set
[0047] Step 5: Establish a linear system of equations based on the constructed matrix and subarrays in, represents,the weight parameter of the model, which determines the influence of the,specification financial characteristics on the target variable;
[0048] Step 6: Solve the linear equations by gradient descent method and convert the weight matrix obtained by the solution into The final weight matrix is obtained by averaging
[0049] Step 7: Processing of new data For a new input matrix X * , use the method in step 3 to construct its submatrix Ensure that the new data has the same structure and properties as the training data and predict the result Y * =X * W * , where W * is the final weight matrix;
[0050] Step 8: Evaluate the prediction results. The specific method is to use the mean square error or mean absolute error to measure the gap between the predicted value and the true value. If the evaluation result shows that the prediction accuracy of the model is not high, you can adjust the feature selection method or add new features.
[0051] Please refer to the attached Figure 1 In step 5, a linear equation system is constructed based on the constructed new matrix Xa and the corresponding target variable vector Yi. This equation system is the core of the model. It expresses the relationship between the features and the target variables in mathematical form. The linear equation system is constructed to solve the weight parameters, that is, to find a set of optimal weight values so that the prediction error of the model on the training data is minimized. In step 6, the gradient descent method is used to solve the linear equation system. This is because the gradient descent method performs well in processing large-scale data sets and complex models. By continuously iterating and adjusting the weight parameters, the gradient descent method can gradually approach the optimal solution. The specific calculation steps are as follows:
[0052] Assume the loss function is in‖*‖ 2 Represents the square of the two norm. In the gradient descent method, the weight vector W is updated at each iteration. i The formula is Where α is the learning rate, is the loss function in W i (t) The gradient at , and initialize the weight vector W i (0) , and then repeat the above calculation steps until the loss function converges or reaches the maximum number of iterations, and obtain multiple weight matrices W i , and finally get the weight matrix Step seven takes over the work of data preparation, feature selection, solving linear equations and weight matrix averaging in the previous steps, and also provides a basis for the result evaluation of step eight. If the prediction result is inaccurate, the feature selection method can be adjusted or new features can be added through the feedback mechanism to further optimize the model and improve its prediction accuracy.
[0053] In this embodiment, steps five, six, seven and eight constitute an iterative optimization cycle. In this cycle, we continuously adjust feature selection, construct linear equations, solve weight parameters, process new data and evaluate prediction results. Through continuous iteration and optimization, we can gradually improve the prediction accuracy and robustness of the model. When we compare the predicted results with the real data, we can see the gap between the two. The definition of low accuracy is based on the data scale and field. For example, if the prediction target is stock price, the price range is between 0 and 1,000 yuan. For a simple short-term prediction model, if the MSE reaches hundreds or even thousands, it usually means that the model accuracy is poor, but if it is long-term In cases where the prediction or data fluctuations themselves are very large, the acceptable range of MSE may be wider. When predicting bond yields, its value range may be around -10% to 10%. If the MSE exceeds 1%, assuming that the data is standardized, it may mean that the model has a large room for improvement. When in use, by processing new data, the submatrix can be constructed using the same method as the training data to ensure that the new data has the same structure and properties as the training data, so that the model can continue to adapt to the ever-changing financial market environment. When the prediction results are inaccurate, the model can be improved by adjusting the feature selection method or adding new features. This feedback mechanism enables the model to continuously learn and optimize to improve the accuracy of predictions.
[0054] When in use, by processing new data, the submatrix can be constructed using the same method as the training data, ensuring that the new data has the same structure and properties as the training data, so that the model can continue to adapt to the ever-changing financial market environment. When the prediction results are inaccurate, the model can be improved by adjusting the feature selection method or adding new features. This feedback mechanism enables the model to continuously learn and optimize, thereby improving prediction accuracy.
[0055] Working principle:
[0056] First, data collection and preprocessing are carried out. Data crawler technology is used to collect data from multiple big data sources in the financial field, including stock exchange transaction data, financial news media platform information, financial market-related discussions on social media, and internal database information of financial institutions. The collected data covers stock price data such as opening price, closing price, highest price, lowest price and trading volume, different types of bond prices and foreign exchange rate data, etc. Then, the collected data is processed to remove noise, outliers and missing values, so as to ensure the quality and reliability of the data and lay a solid foundation for subsequent steps. Then, the matrix is constructed to construct the processed data into a matrix pair (X i , Y i ), then, for each X i For feature selection, in addition to selecting moving average, relative strength index, GDP growth rate, etc., other features related to the financial market such as trading volume, price-earnings ratio, price-to-book ratio, financial ratio, etc. can also be considered. Then X i It is defined as a rank n submatrix to reduce the complexity of the model, and then a new matrix is constructed and a system of equations is established to construct a new matrix Xa, and the corresponding conditions are set. Based on the constructed matrix and submatrix, a system of linear equations is established. The weight parameters involved determine the degree of influence of financial characteristics on the target variable. After that, the system of equations and the average weight matrix are solved, and the linear equations are solved by the gradient descent method to obtain the final weight matrix. Finally, new data and model optimization are processed. For new data, the submatrix is constructed using the same method as the training data to ensure that the new data has the same structure and properties as the training data, and then the results are predicted. The prediction results are evaluated by the mean square error or the mean absolute error. If the accuracy is not high, adjust the feature selection method or add new features. Through this feedback mechanism, the model can continuously learn and optimize to adapt to changes in the financial market.
[0057] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0058] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
[0059] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A high-dimensional data prediction and processing method based on big data technology, characterized in that: The prediction method specifically comprises the following steps: Step 1: Use data crawler technology to collect data from big data sources in the financial field, including trading data from stock exchanges, financial news media platforms, discussions related to financial markets on social media, internal databases of financial institutions, etc., and ensure the quality and reliability of the data by removing noise, outliers and missing values; Step 2: Construct the overall data into a matrix pair (X i , Y i ) in the form of i is a matrix of financial data with a dimension of m×n, where m can represent the number of features of financial data, such as technical indicators and macro indicators of different stocks, and n can represent the number of samples. i is the corresponding target variable vector and m>n; Step 3: For each X in the data i Perform feature selection, including but not limited to selecting moving average, relative strength index, and GDP growth rate, and then define X i Become a sub-matrix The submatrix is a rank n matrix; Step 4: Construct a new matrix and name it Xa, in the form of (Xa, I(mn)(mn)), where I(mn)(mn) is an identity matrix, and set Step 5: Establish a linear system of equations based on the constructed matrix and subarrays in, represents,the weight parameter of the model, which determines the influence of the,specification financial characteristics on the target variable; Step 6: Solve the linear equations by gradient descent method and convert the weight matrix obtained by the solution into The final weight matrix is obtained by averaging Step 7: Processing of new data For a new input matrix X * , use the method in step 3 to construct its submatrix Ensure that the new data has the same structure and properties as the training data and predict the result Y * =X * W * , where W * is the final weight matrix; Step 8: Evaluate the prediction results. The specific method is to use the mean square error or mean absolute error to measure the gap between the predicted value and the true value. If the evaluation result shows that the prediction accuracy of the model is not high, you can adjust the feature selection method or add new features.
2. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: The data collected by the financial data collection mentioned in step 1 include, but are not limited to, stock price data (opening price, closing price, highest price, lowest price and trading volume), prices of different types of bonds and foreign exchange rate data.
3. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: The number of features m in step 2 will be much larger than the number of samples n. This is because financial data often has high-dimensional characteristics. Therefore, m can represent a large number of financial feature variables, and n can represent different time points or transaction instances. Therefore, when analyzing stock market data, a large number of technical indicators and macroeconomic variables will appear as features, and the number of samples is the daily trading data or the observation value within a specific time period.
4. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: The feature selection mentioned in the step three aims to select the most informative features from the original data set to improve the performance of the model and the accuracy of prediction. For example, features such as moving average, relative strength index and GDP growth rate can be selected. The selection of these features is based on their influence and relevance to the financial market or specific stocks. In addition to these features, other features related to the financial market can also be considered, such as trading volume, price-earnings ratio, price-to-book ratio, financial ratios, etc. The construction of the submatrix helps to reduce the complexity of the model. In addition, feature selection and submatrix construction is an iterative and optimization process. In subsequent steps, if the evaluation results show that the prediction accuracy of the model is not high, the feature selection method can be adjusted or new features can be added to improve the model.
5. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: In the step 4, a new matrix Xa is constructed based on the feature-based submatrix. In the prediction result evaluation in step 8, if the prediction accuracy of the model is not high, the new matrix construction method in step 4 can be adjusted, including increasing or decreasing the dimension of the unit matrix and changing the feature selection strategy to improve the prediction accuracy and robustness of the model.
6. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: In step five, a linear equation group is constructed based on the constructed new matrix Xa and the corresponding target variable vector Yi. This equation group is the core of the model, which expresses the relationship between the features and the target variables in mathematical form. The linear equation group is constructed to solve the weight parameters, that is, to find a set of optimal weight values so that the prediction error of the model on the training data is minimized.
7. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: In step 6, the gradient descent method is used to solve the linear equations. This is because the gradient descent method performs well in processing large-scale data sets and complex models. By continuously iterating and adjusting weight parameters, the gradient descent method can gradually approach the optimal solution. The specific calculation steps are as follows: Assume the loss function is in‖*‖ 2 Represents the square of the two norm. In the gradient descent method, the formula for updating the weight vector Wi at each iteration is Where α is the learning rate, The loss function is The gradient at and initialize the weight vector Then repeat the above calculation steps until the loss function converges or reaches the maximum number of iterations to obtain multiple weight matrices W i , and finally get the weight matrix 8. The high-dimensional data prediction and processing method based on big data technology according to claim 1 is characterized in that: The step seven takes over the work of data preparation, feature selection, solving linear equations and weight matrix averaging in the previous steps, and at the same time provides a basis for the result evaluation of step eight. If the prediction result is inaccurate, the feature selection method can be adjusted or new features can be added through the feedback mechanism to further optimize the model and improve its prediction accuracy.