Prediction method and device based on principal component regression and nonvolatile storage medium
By introducing Huber loss function and SCAD punishment function in the principal component regression model, and combining ADMM algorithm and Gauss Seidel iteration method, the lack of robustness and prediction accuracy of the principal component regression method is solved, and more efficient features are achieved Extract and predict effects.
Patent Information
- Application Number
- CN202411187339.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-11-08
AI Technical Summary
The existing principal component regression methods have shortcomings in terms of robustness, prediction accuracy and interpretability, and it is difficult to effectively deal with the impact of multicollinearity problems and outliers.
The target principal component regression model is adopted to improve the robustness and prediction accuracy of the model through the combination of total loss function and total penalty function, including Huber loss function and SCAD penalty function, and solve sparseness through ADMM algorithm and Gauss Seidel iterative method. Principal component regression coefficient.
The robustness and prediction accuracy of the principal component regression model are improved, the noise immunity and interpretability of the model are enhanced, and the impact of the direction outliers of the dependent variable on the model error is reduced.
Smart Images

Figure CN118916689A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of machine learning, and more specifically, to a prediction method, device and non-volatile storage medium based on principal component regression. Background Art
[0002] In recent years, dimensionality reduction and feature extraction of data, discovering key variables while reducing dimensions and minimizing information loss have become increasingly important in the field of high-dimensional statistics. Quantifying events and their corresponding causal relationships can be applied to scenarios such as drug clinical trials, engineering safety monitoring, sales research, etc., especially in prediction scenarios involving multiple features. Principal Component Regression (PCR) is such a multivariate statistical method that can effectively reduce the dimension of variables, has excellent properties in dealing with multicollinearity problems of variables, and reveals the intrinsic relationship between independent variables and dependent variables. However, the penalty function and loss function of the existing principal component regression method in related technologies cannot guarantee the robustness, prediction accuracy and interpretability of the principal component regression model. There is an urgent need to optimize the existing principal component regression method to improve the robustness, prediction accuracy and interpretability of the principal component regression model.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present application provide a prediction method, device and non-volatile storage medium based on principal component regression to at least solve the technical problems of low robustness, prediction accuracy and interpretability of the principal component regression model of the existing principal component regression method in the related art.
[0005] According to one aspect of an embodiment of the present application, a prediction method based on principal component regression is provided, comprising: obtaining original data to be predicted; predicting the original data using a target principal component regression model to obtain predicted data, wherein the target principal component regression model includes an objective function consisting of a total loss function and a total penalty function, the total loss function includes a first loss function representing the principal component regression and a second loss function used to represent the principal component decomposition of the original data, and the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient.
[0006] In some embodiments of the present application, the method also includes: obtaining predicted data of the original data, and obtaining the intercept term of the estimated coefficient, the data matrix of the original data, the loading matrix, and the principal component regression coefficient; adjusting the Huber loss function based on the predicted data, the intercept term of the estimated coefficient, the data matrix, the loading matrix, and the principal component regression coefficient to obtain a first loss function.
[0007] In some embodiments of the present application, the first loss function is expressed by the following formula:
[0008]
[0009] Among them, L reg (γ0,γ,B) represents the first loss function, n represents the number of features of the original data, l τ (y i -γ0-x i T Bγ) is the Huber loss function, y i represents the i-th predicted data, γ0 is the intercept term of the estimated coefficient, x i represents the i-th original data, B represents the loading matrix, and γ represents the principal component regression coefficient.
[0010] In some embodiments of the present application, the Huber loss function is expressed by the following formula:
[0011]
[0012] Among them, l τ (y i -γ0-x i T Bγ) represents the Huber loss function, and τ is the threshold parameter of the Huber loss function.
[0013] The above Huber loss function has different effects in different situations. τ>0 represents the hyperparameter that controls the Huber function. When τ→∞, l τ (y i -γ0-x i T Bγ) degenerate ordinary least squares loss; when τ→0, l τ (·y i -γ0-x i T Bγ) degenerates into absolute value loss, where the value of τ depends on how to define the size of the outlier in the actual problem. Huber loss is a combination of square loss and absolute loss. It overcomes the shortcomings of square loss and absolute loss. It not only makes the loss function have continuous derivatives, but also uses the characteristic that the MSE gradient decreases with the error to obtain a more accurate minimum value.
[0014] The second loss function is expressed by the following formula:
[0015]
[0016] Among them, L PCA(A, B) represents the second loss function, n represents the number of features of the original data, y i represents the i-th predicted data, X represents the data matrix composed of the original data, B represents the loading matrix, ω represents the weight coefficient, F represents the norm, and A represents the principal component matrix.
[0017] In some embodiments of the present application, the first penalty function is expressed by the following formula:
[0018] P1(B;λ B )=λ B ‖B‖1
[0019] Among them, P1(B;λ B ) represents the first penalty function, λ B represents the regularization parameter, and B represents the loading matrix.
[0020] In some embodiments of the present application, the second penalty function is expressed by the following formula:
[0021]
[0022] Among them, P2(γ,λ) represents the second penalty function, λ represents the regularization parameter, represents the SCAD penalty function, γ represents the principal component regression coefficient, and γ j is the jth principal component regression coefficient.
[0023] In some embodiments of the present application, the SCAD penalty function is expressed by the following formula:
[0024]
[0025] in, represents the SCAD penalty function, I represents the characteristic function, λ represents the tuning parameter, μ represents the input of the SCAD penalty function, and a represents the preset parameter.
[0026] According to another aspect of an embodiment of the present application, a prediction device based on principal component regression is also provided, including: an acquisition module for acquiring original data to be predicted; a prediction module for using a target principal component regression model to predict the original data to obtain predicted data, wherein the target principal component regression model includes an objective function consisting of a total loss function and a total penalty function, the total loss function includes a first loss function representing the principal component regression and a second loss function used to represent the principal component decomposition of the original data, and the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient.
[0027] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned prediction methods based on principal component regression.
[0028] According to another aspect of an embodiment of the present application, an electronic device, a memory and a processor are also provided, wherein the processor is used to run a program stored in the memory, wherein when the program is run, any one of the above-mentioned prediction methods based on principal component regression is executed.
[0029] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement any of the above-mentioned prediction methods based on principal component regression.
[0030] In an embodiment of the present application, the original data to be predicted is obtained; the target principal component regression model is used to predict the original data to obtain predicted data, wherein the target principal component regression model includes a target function composed of a total loss function and a total penalty function, the total loss function includes a first loss function representing the principal component regression and a second loss function used to represent the principal component decomposition of the original data, the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient, the original data is predicted by the target principal component regression model to obtain predicted data, wherein the total loss function and the total penalty function improve the robustness of the model, thereby improving the prediction accuracy and interpretability, and reducing the error influence of the outliers in the direction of the dependent variable on the model, thereby solving the technical problems of low robustness, prediction accuracy and interpretability of the principal component regression model of the existing principal component regression method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0032] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a prediction method based on principal component regression according to an embodiment of the present application;
[0033] Figure 2 is a flow chart of a prediction method based on principal component regression according to an embodiment of the present application;
[0034] Figure 3 It is a schematic diagram of a process of robust sparse principal component regression solution according to an embodiment of the present application;
[0035] Figure 4 It is a structural schematic diagram of a prediction device based on principal component regression according to an embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0038] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0039] Principal Component Regression (PCR): is a statistical method used to reduce the dimension of features in regression analysis. It combines the concepts of principal component analysis and linear regression to simplify the model by extracting the main features in the data while maintaining the accuracy of the prediction.
[0040] Sparse Principal Component Regression Alternating Direction Method of Multipliers (SPCR-ADMM) is an algorithm that combines Sparse Principal Component Regression (SPCR) and Alternating Direction Method of Multipliers (ADMM). This algorithm is mainly used to deal with the dimensionality reduction problem of high-dimensional data while maintaining the sparsity of the data. Sparse Principal Component Regression SPCR is a method for dimensionality reduction and feature extraction. It is achieved by adding sparsity constraints on the basis of principal component analysis. This method can effectively extract representative features while maintaining the sparsity of the data, which helps to improve the interpretability and generalization ability of the model. Alternating Direction Method of Multipliers ADMM is an optimization algorithm used to solve optimization problems with complex constraints. It decomposes the original problem into two sub-problems and solves the two sub-problems alternately until convergence. ADMM has a wide range of applications in many fields, such as image processing, machine learning, etc.
[0041] Huber Loss Function (Huber Loss Function): A loss function used for regression problems. It is robust when dealing with outliers. The Huber Loss Function combines the advantages of Mean Squared Error (MSE) and Mean Absolute Error (MAE), allowing the model to balance the sensitivity to outliers and the fit to other data points when fitting data. The Huber Loss Function is widely used in various regression problems in machine learning and deep learning, especially when there are outliers or noise in the data. It can help the model fit the data better and improve the accuracy of predictions.
[0042] Least Absolute Shrinkage and Selection Operator (Lasso) penalty: is a regularization technique commonly used in statistics and machine learning, especially in linear regression and logistic regression problems. The purpose of the Lasso penalty is to reduce the complexity of the model, prevent overfitting, and facilitate feature selection. The Lasso penalty achieves this by adding a regularization term to the loss function.
[0043] Convex Penalty Function: A function used to penalize non-convex or non-ideal behavior in an optimization problem. Convex penalty functions are often used for regularization problems to ensure that the solution to the optimization problem satisfies certain properties, such as sparsity, smoothness, or stability.
[0044] Alternating Least Absolute Shrinkage and Selection Operator (ALASSO): is a statistical method mainly used for variable selection and regularization. It combines the sparsity property of LASSO penalty and the robustness of least squares method.
[0045] In the related art, the penalty function and loss function of the existing principal component regression method in the related art cannot guarantee the robustness, prediction accuracy and interpretability of the principal component regression model, and it is urgent to optimize the existing principal component regression method. Therefore, there is a problem that the principal component regression model of the existing principal component regression method in the related art has low robustness, prediction accuracy and interpretability. In order to solve this problem, a relevant solution is provided in the embodiment of the present application.
[0046] According to an embodiment of the present application, a method embodiment of a prediction method based on principal component regression is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0047] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a prediction method based on principal component regression. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0048] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0049] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the prediction method based on principal component regression in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned prediction method based on principal component regression. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0050] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0051] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .
[0052] In the above operating environment, the present application embodiment provides a flow chart of a prediction method based on principal component regression, such as Figure 2 As shown, the method comprises the following steps:
[0053] Step S202, obtaining original data to be predicted.
[0054] In the technical solution provided in step S202, the raw data to be predicted can be data of various prediction scenarios, which are not specifically limited in this application. For example, it can be applied to drug clinical trials, engineering safety monitoring, sales research, prediction of the real power of energy power plants based on air-conditioning system operation, prediction of concrete strength, prediction of the median value of owner-occupied houses, etc., especially in prediction scenarios involving multiple features. The raw data to be predicted can include the data matrix corresponding to the raw data, the data label and the number of principal components.
[0055] Step S204, using the target principal component regression model to predict the original data to obtain predicted data, wherein the target principal component regression model includes an objective function consisting of a total loss function and a total penalty function.
[0056] In the technical solution provided in step S204, the total loss function includes a first loss function representing principal component regression and a second loss function for representing principal component decomposition of the original data, the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient, and the total loss function can quantify the error of the model in the prediction process. By minimizing the total loss function, the optimal model parameters can be found, thereby improving the prediction accuracy of the model; the loss function contains a regularization term, which helps to achieve the sparsity of the model. Sparsity means that many coefficients in the model may be zero, thereby simplifying the model structure and improving the interpretability of the model. By minimizing the loss function, the optimal model parameters can be found to achieve the best fit of the data; the total penalty function is usually used to encourage the sparsity of the model in robust sparse principal component regression. This means that the model will tend to select fewer features (variables) to build a regression model, which helps to improve the interpretability of the model and reduce the risk of overfitting. Robust sparse principal component regression pays special attention to robustness to outliers and noise. The total penalty function can help the model remain stable in the face of these outliers and reduce their impact on model performance. Through the total penalty function, the model can automatically select the features that have the greatest impact on the prediction results, which helps to simplify the model, improve the interpretability of the model, and may improve the prediction accuracy. In robust sparse principal component regression, the total penalty function is usually combined with the objective function to form an optimization problem. Solving this optimization problem can obtain the parameters of the model, which reflect the main structures and patterns in the data.
[0057] In the technical solution provided in step S204, the target principal component regression model is a robust sparse principal component regression (RSPCR), and the objective function consisting of a total loss function and a total penalty function can be expressed by the following formula:
[0058]
[0059] sB T B=I k×k ,
[0060] Among them, B T B represents the product of the transpose of the loading matrix B of the data X and itself, I k×k represents the identity matrix, stB T B=I k×k The overall meaning is B T B is an orthogonal matrix, L reg (γ0,γ,B) represents the loss function of principal component regression (i.e., the first loss function mentioned above), L PCA (A, B) (i.e., the second loss function) represents the loss function of principal component decomposition of data X, P1(B; λ B )(i.e., the first penalty function mentioned above) represents the penalty function of principal component regression on the load matrix B, P2(γ,λ) (i.e., the second penalty function mentioned above) represents the penalty function of principal component regression on the regression coefficient, A=(α1,…,α k )∈R n×k represents the n×k principal component matrix, B=(β1,β2,…,β k )∈R p×k represents the loading matrix, γ0 represents the intercept term of the estimated coefficient, γ=(γ1,…,γ k ) represents the principal component regression coefficient, λ β and λ are regularization parameters, both of which are positive. Data X is the original data mentioned above, X=(x1,x2,…,x n ) T ∈R n×p is an n×p data matrix.
[0061] The principal component matrix is a linear combination of the original data. These linear combinations are the directions of greatest variability in the data. In PCR, the principal component matrix is often used to replace the original predictor variables; the loading matrix is a matrix that maps the original variables to the principal components. Each of its columns represents the weight of the original variable on the corresponding principal component; in regression analysis, the estimated coefficients are the parameters used to predict the response variable. These coefficients are for the principal components rather than the original variables; the intercept term is the value of the response variable predicted by the model when all predictor variables (i.e., predicted data) are zero.
[0062] Obtain the predicted data of the original data, as well as the intercept term of the estimated coefficient, the data matrix of the original data, the load matrix, and the principal component regression coefficient; adjust the Huber loss function based on the predicted data, the intercept term of the estimated coefficient, the data matrix, the load matrix, and the principal component regression coefficient to obtain the first loss function, which is expressed by the following formula:
[0063]
[0064] Among them, L reg (γ0,γ,B) represents the first loss function, n represents the number of features of the original data, l τ (y i -γ0-x i T Bγ) is the Huber loss function, y i represents the i-th predicted data, γ0 is the intercept term of the estimated coefficient, x i represents the i-th original data, B represents the loading matrix, and γ represents the principal component regression coefficient.
[0065] The Huber loss function is expressed by the following formula:
[0066]
[0067] Among them, l τ (y i -γ0-x i T Bγ) represents the Huber loss function, and τ is the threshold parameter of the Huber loss function.
[0068] The second loss function is expressed by the following formula:
[0069]
[0070] Among them, L PCA (A, B) represents the second loss function, n represents the number of features of the original data, y i represents the i-th predicted data, X represents the data matrix composed of the original data, B represents the loading matrix, ω represents the weight coefficient, F represents the norm, and A represents the principal component matrix.
[0071] The above F represents the norm, which is defined as:
[0072]
[0073] That is, the square root of the sum of the squares of each number in the data matrix.
[0074] The penalty function (i.e., the first penalty function) of the principal component regression on the load matrix B is expressed by the following formula:
[0075] P1(B;λ B )=λ B ‖B‖1
[0076] Among them, P1(B;λ B ) represents the first penalty function, λ B represents the regularization parameter, and B represents the loading matrix.
[0077] The penalty function of the principal component regression on the load matrix B (i.e., the second penalty function) is expressed by the following formula:
[0078]
[0079] Among them, P2(γ,λ) represents the second penalty function, λ represents the regularization parameter, represents the SCAD penalty function, γ represents the principal component regression coefficient, and γ j is the jth principal component regression coefficient.
[0080] The SCAD penalty function in the second penalty function is expressed by the following formula:
[0081]
[0082] in, represents the SCAD penalty function, I represents the characteristic function, λ represents the tuning parameter, μ represents the input of the SCAD penalty function, and a represents a preset parameter (for example, set to a=3.7).
[0083] Since the above SCAD penalty function is a non-concave function, in order to effectively solve the model, the local linear approximation (LLA) is used to asymptotically transform the SCAD penalty function into the following form:
[0084]
[0085] in, yes The kth component of yes The first-order derivative of is defined as:
[0086]
[0087] Where μ>0, μ represents the input of the SCAD penalty function, a represents the preset parameter, λ represents the regularization parameter, which is a positive value, and I represents the indicative function.
[0088] In the technical solution provided in step S204, the target principal component regression model (i.e., the robust sparse principal component regression model) needs to be trained before prediction. For example, the model training can be performed in the following manner and the robust sparse principal component regression model can be solved using the ADMM algorithm:
[0089] Input the data matrix corresponding to the original data, the data label and the number of principal components (corresponding to the original data to be predicted above), and determine the augmented Lagrangian function of the objective function as follows:
[0090]
[0091] sB T B=I k×k
[0092] in, represents the augmented Lagrangian function, ρ1, ρ2 and ρ3 are penalty parameters, B1, B, B0 are loading matrices, Λ1, Λ2 and λ3 are dual variables.
[0093] The steps of the ADMM algorithm to solve the robust sparse principal component regression model are:
[0094] Step 1: Select parameter ω, Huber hyperparameter τ, regularization parameter λ B and λ, non-negative penalty parameters ρ1, ρ2, and ρ3.
[0095] Step 2: Initialize parameters (This is the initial parameter without iteration).
[0096] Step 3: Repeat steps 4 to 11 for k = 0, 1, 2, 3, ... until convergence, and output the sparse matrix and regression coefficients.
[0097] Step 4: Update B1. is B1 after the k+1th iteration, and the following steps are similar:
[0098]
[0099] in,
[0100]
[0101] because:
[0102]
[0103] in is a symbolic function, I(|y i -γ0-x i T B1γ (k) ·) is the indicative function.
[0104] because:
[0105]
[0106] Where C is a constant independent of B1, then:
[0107]
[0108] The solution is obtained by using the Gauss-Seidel algorithm therefore Indicates that the solution obtained by step k is To update the B1 value of step k+1, that is
[0109] Step 5: Update of B:
[0110]
[0111] Among them, C is independent of B and is a constant.
[0112]
[0113] The solution can be obtained:
[0114] B (k+1) =PQ T ,
[0115] Among them, the P and Q matrices are B (k+1) The singular value decomposition matrix of , P is the p×p left singular vector matrix, and Q is the k×k right singular vector matrix. It comes from the following singular value decomposition formula:
[0116]
[0117] Step 6: Update of B0:
[0118]
[0119] The update problem is solved using the proximal gradient descent algorithm, and its solution is:
[0120]
[0121] in, Indicates B (k+1) The element in row i and column j, express The element in row i and column j, express The element in row i and column j, for The element in row i and column j, means for The element in row i and column j It refers to the explicit solution form of linear regression under Lasso regression, i.e. L_1 penalty function, under orthogonal conditions. It depends on the sign of μ and the size of |μ|-α and 0.
[0122] Step 7: Update of A:
[0123]
[0124] sB T B=I k
[0125] The right side of the equation is differentiated with respect to A, and the first-order derivative is O, so we get
[0126] A (k+1) =XB (k+1)
[0127] Step 8: Update of γ:
[0128]
[0129] make γ * is the regression coefficient calculated in the k-th iteration,
[0130] Solve the following Huber+ridge regression model to obtain γ * :
[0131]
[0132] Here, the solution is optimized using the Locally Adaptive Mixture Model (LAMM) algorithm to obtain γ * , and then return to the formula In, we get:
[0133] Step 9: Update of γ1:
[0134]
[0135] is equivalent to:
[0136]
[0137] in, γ 1i , and They are γ1, γ (k+1) and The i-th component of , when the hyperparameters λ and a of the SCAD penalty function are determined, About Gamma 1i Derivative:
[0138]
[0139] Since P2(γ 1i ,λ) is symmetric about the y-axis, without loss of generality,γ 1i >0,
[0140]
[0141] Easy to understand Due to symmetry another solution is Substituting the two solutions into The smallest solution is denoted by therefore
[0142] in
[0143] Step 10: Update of γ0:
[0144]
[0145] Denoted as f(γ0)=0;
[0146]
[0147] f(γ0) is a first-order continuous function of γ0. It is easy to see that there exists The derivative of f(γ0) is negative. According to the zero point existence theorem, f(γ0) has a unique zero point. The zero point can be easily solved using the bisection method.
[0148] Step 11: Update of Λ1, Λ2 and λ3:
[0149]
[0150] In step S204, the data matrix corresponding to the original data required for training the target principal component regression model, the data label and the number of principal components can be loaded into the memory. For example, the data matrix corresponding to the original data, the data label and the original data of the number of principal components can be loaded from the non-volatile memory to the volatile memory, so that the processor runs the target principal component regression model. The data matrix corresponding to the original data, the data label and the original data of the number of principal components refer to unprocessed data, and usually include parameters and structural data of the target principal component regression model. The structural data can be a calculation relationship based on the parameters, such as a forward propagation calculation relationship between intermediate layers. Specifically, the structural data can include code related to the structure of the target principal component regression model, such as code for performing related calculations between intermediate layers.
[0151] In one embodiment, an area for training the target principal component regression model can be divided in the memory, which can include a structure data storage area and a parameter storage area. The structure data storage area is used to store structure-related codes, and the parameters referenced by it can point to the address of specific parameters in the parameter storage area through a pointer. During the training process of the target principal component regression model, it may be necessary to frequently update the parameters, and the parameter values in the parameter storage area can be updated.
[0152] Through the above steps, the target principal component regression model (i.e., the robust sparse principal component regression model) in the embodiment of the present application can be realized by integrating the optimal low-rank approximation of the data matrix and the principal component regression into one model, i.e., the first loss function L reg (γ0,γ,B) and the second loss function L PCA (A, B) minimization is put in one equation; Since the condition for sparse principal component extraction is that the load matrix is orthogonal, the sparse principal component load matrix estimation condition is relaxed, and the sparse principal component estimation constraint condition is relaxed to a convex set, so that the low-rank approximation of the data matrix is transformed into a convex optimization problem. The robust sparse principal component regression model in the embodiment of the present application focuses on studying the robust properties of the model. When the data contains heavy-tailed errors, whether the principal component regression model can have a higher prediction accuracy, therefore, the Huber loss function is used in the selection of the loss function of the principal component regression. In order to make the principal component regression coefficient interpretable, the penalty function for the principal component regression coefficient in the objective function adopts the SCAD penalty function, so that the principal component regression coefficient becomes sparse. In the above-mentioned solution steps, the core algorithm uses the alternating multiplier algorithm, the algorithm iteratively solves each variable and the dual variable, the principal component regression coefficient solution adopts the LAMM algorithm approximate solution, and the sparse matrix solution adopts the Gauss-Seidel iteration method.
[0153] The objective function of the target principal component regression model (i.e., the robust sparse principal component regression model) in the embodiment of the present application replaces the ordinary least squares loss of the principal component regression loss function with the Huber loss function. When the noise of the data is thick-tailed and heavy-tailed distributions, the deviation of a single data is balanced by the hyperparameters of the Huber loss function. When the data deviation is small, the ordinary least squares loss is taken, and when the data deviation is large, the absolute value loss is taken, thereby improving the noise resistance of the model. A sparse principal component regression model is established using the robustness loss function to improve the robustness, prediction accuracy, and interpretability of the principal component regression model. The ADMM algorithm is optimized, the sparse matrix is iterated using Gauss-Seidel, and the iteration of the loss function is approximated using the LLA algorithm. The algorithm converges while improving computational efficiency.
[0154] Table 1 and Table 2 are respectively the characteristics and sample size statistics of the data in the embodiment of the present application under the low-dimensional real data set and the high-dimensional real data set:
[0155] Table 1
[0156]
[0157] The number of principal components is calculated by pre-calculating the cumulative variance contribution rate of each data set. If the cumulative variance contribution rate exceeds 90%, it can be selected as the number of principal components and input into the model. The data comes from the UCI data set. Housing is the UCI Boston housing data set, which aims to predict the median value of owner-occupied houses through 13 features; Energy is the UCI energy data set, which is used to predict the real power of energy power plants based on air-conditioning system operation; Forest is the UCI forest data set, which predicts the forest cover type to which the sample belongs through 10 basic features; Concrete is the UCI concrete data set, which predicts the strength of concrete through 8 features.
[0158] Table 2
[0159]
[0160] These five data sets are all genetic data sets. Breast Cancer1 represents breast cancer data set 1, Breast Cancer2 represents breast cancer data set 2, DLBCL represents diffuse large B-cell lymphoma, AML represents acute myeloid leukemia, and NSCLC represents non-small cell lung cancer. The dependent variables are continuous indicators, so they can all be used for regression prediction.
[0161] The following Tables 3 and 4 respectively use the method in the embodiment of the present application and the classical principal component regression model to perform the results of the low-dimensional real data set and the high-dimensional data set in Table 1 and Table 2. Table 3 is the MSE mean (variance) of the real data set results, and Table 4 is the MSE mean (variance) of the real data set results:
[0162] Table 3
[0163]
[0164]
[0165] Table 4
[0166]
[0167] After obtaining the real data and performing basic preprocessing on the data, the number of principal components is determined by accumulating the variance contribution rate. The RSPCR model of the application embodiment and the comparative SPCR-ADMM model, the Alternating Splitting Conjugate Primal-Dual Algorithm (ASPCR), and the Sparse Primal-Dual Optimization (SPCR) model all take the same number of principal components. It can be seen that the RSPCR method of the application embodiment performs best.
[0168] The present application also provides a schematic diagram of a robust sparse principal component regression solution process, such as Figure 3 As shown,
[0169] Input data (i.e., the original data to be predicted is obtained as mentioned above), determine whether the data is standard, and perform parameter initialization settings (i.e., the above-mentioned initialization parameters) when the data is standard. If the data is not standard, preprocess the data (for example, perform data standardization), and then perform parameter initialization settings (i.e., the above-mentioned initialization parameters). Then, iterate the algorithm value to converge and output the sparse matrix and regression coefficients (corresponding to the steps of solving the robust sparse principal component regression model by the above-mentioned ADMM algorithm).
[0170] The present application embodiment provides a schematic diagram of the structure of a prediction device for principal component regression, such as Figure 4 As shown, it includes an acquisition module 402 and a prediction module 404 .
[0171] The acquisition module 402 is used to acquire the raw data to be predicted. The raw data to be predicted can be data of various prediction scenarios, which are not specifically limited in this application. For example, it can be applied to drug clinical trials, engineering safety monitoring, sales research, prediction of the real power of energy power plants based on air conditioning system operation, prediction of concrete strength, prediction of the median value of owner-occupied houses, etc., especially in prediction scenarios involving multiple features. The raw data to be predicted can include the data matrix corresponding to the raw data, the label of the data and the number of principal components.
[0172] The prediction module 404 is used to use the target principal component regression model to predict the original data to obtain predicted data, wherein the target principal component regression model includes an objective function consisting of a total loss function and a total penalty function, the total loss function includes a first loss function representing the principal component regression and a second loss function used to represent the principal component decomposition of the original data, and the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient.
[0173] The target principal component regression model mentioned in the prediction module 404 is a robust sparse principal component regression (Robust Sparse Principal Component Regression, referred to as RSPCR), and the objective function composed of the total loss function and the total penalty function can be expressed by the following formula:
[0174]
[0175] sB T B=I k×k ,
[0176] Among them, B T B represents the product of the transpose of the loading matrix B of the data X and itself, I k×k represents the identity matrix, stB T B=I k×k The overall meaning is B T B is an orthogonal matrix, L reg (γ0,γ,B) represents the loss function of principal component regression (i.e., the first loss function mentioned above), L PCA (A, B) (i.e., the second loss function) represents the loss function of principal component decomposition of data X, P1(B; λ B )(i.e., the first penalty function mentioned above) represents the penalty function of principal component regression on the load matrix B. P2(γ,λ) (i.e., the second penalty function mentioned above) represents the penalty function of principal component regression on the regression coefficient, A=(α1,…,α k )∈R n×k represents the n×k principal component matrix, B=(β1,β2,…,β k )∈R p×k represents the loading matrix, γ0 represents the intercept term of the estimated coefficient, γ=(γ1,…,γ k ) represents the principal component regression coefficient, λ β and λ are regularization parameters, both are positive.
[0177] The prediction module 404 is also used to obtain the prediction data of the original data, and obtain the intercept term of the estimated coefficient, the data matrix of the original data, the loading matrix, and the principal component regression coefficient; according to the prediction data, the intercept term of the estimated coefficient, the data matrix, the loading matrix, and the principal component regression coefficient, the Huber loss function is adjusted to obtain a first loss function represented by the following formula:
[0178]
[0179] Among them, L reg (γ0,γ,B) represents the first loss function, n represents the number of features of the original data, l τ (y i -γ0-x i T Bγ) is the Huber loss function, y i represents the i-th predicted data, γ0 is the intercept term of the estimated coefficient, x i represents the i-th original data, B represents the loading matrix, and γ represents the principal component regression coefficient.
[0180] The Huber loss function is expressed by the following formula:
[0181]
[0182] Among them, l τ (y i -γ0-x i T Bγ) represents the Huber loss function, and τ is the threshold parameter of the Huber loss function.
[0183] The second loss function is expressed by the following formula:
[0184]
[0185] Among them, L PCA (A, B) represents the second loss function, n represents the number of features of the original data, y i represents the i-th predicted data, X represents the data matrix composed of the original data, B represents the loading matrix, ω represents the weight coefficient, F represents the norm, and A represents the principal component matrix.
[0186] The above F represents the norm, which is defined as:
[0187]
[0188] That is, the square root of the sum of the squares of each number in the data matrix.
[0189] The penalty function (i.e., the first penalty function) of the principal component regression on the load matrix B is expressed by the following formula:
[0190] P1(B;λ B )=λ B ‖B‖1
[0191] Among them, P1(B;λ B ) represents the first penalty function, λ B represents the regularization parameter, and B represents the loading matrix.
[0192] The penalty function of the principal component regression on the load matrix B (i.e., the second penalty function) is expressed by the following formula:
[0193]
[0194] Among them, P2(γ,λ) represents the second penalty function, λ represents the regularization parameter, represents the SCAD penalty function, γ represents the principal component regression coefficient, and γ j is the jth principal component regression coefficient.
[0195] The SCAD penalty function in the second penalty function is expressed by the following formula:
[0196]
[0197] Among them, φ λ (|μ|) represents the SCAD penalty function, I represents the characteristic function, λ represents the tuning parameter, μ represents the input of the SCAD penalty function, and a represents a preset parameter (for example, set to a=3.7).
[0198] It should be noted that the various modules in the above-mentioned prediction device based on principal component regression can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0199] The embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned prediction method based on principal component regression. For example, the original data to be predicted is obtained; the original data is predicted using the target principal component regression model to obtain predicted data, wherein the target principal component regression model includes an objective function composed of a total loss function and a total penalty function, the total loss function includes a first loss function representing the principal component regression and a second loss function for representing the principal component decomposition of the original data, and the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient.
[0200] The embodiment of the present application also provides an electronic device, the electronic device includes a processor, the processor is used to run a program, wherein the above-mentioned prediction method based on principal component regression is executed when the program is running. For example, the original data to be predicted is obtained; the original data is predicted using the target principal component regression model to obtain predicted data, wherein the target principal component regression model includes an objective function composed of a total loss function and a total penalty function, the total loss function includes a first loss function representing the principal component regression and a second loss function used to represent the principal component decomposition of the original data, and the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient.
[0201] According to another aspect of the embodiment of the present application, a computer program product is also provided, including a computer program, which implements the above-mentioned prediction method based on principal component regression when executed by a processor. For example, the original data to be predicted is obtained; the original data is predicted using a target principal component regression model to obtain predicted data, wherein the target principal component regression model includes an objective function composed of a total loss function and a total penalty function, the total loss function includes a first loss function representing principal component regression and a second loss function for representing principal component decomposition of the original data, and the total penalty function includes a first penalty function of the principal component regression on the load matrix and a second penalty function of the principal component regression on the regression coefficient.
[0202] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0203] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0204] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0205] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0206] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0207] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A prediction method based on principal component regression, characterized in that: include: Obtain the raw data to be predicted; The original data is predicted using a target principal component regression model to obtain predicted data, wherein the target principal component regression model includes an objective function consisting of a total loss function and a total penalty function, the total loss function includes a first loss function representing principal component regression and a second loss function used to represent principal component decomposition of the original data, and the total penalty function includes a first penalty function of principal component regression on the load matrix and a second penalty function of principal component regression on the regression coefficient.
2. The method according to claim 1, characterized in that The method further comprises: Obtaining predicted data of the original data, as well as the intercept term of the estimated coefficient, the data matrix of the original data, the loading matrix, and the principal component regression coefficient; The Huber loss function is adjusted according to the predicted data, the intercept term of the estimated coefficient, the data matrix, the loading matrix and the principal component regression coefficient to obtain the first loss function.
3. The method according to claim 2, characterized in that The first loss function is expressed by the following formula: Among them, L reg (γ0,γ,B) represents the first loss function, n represents the number of features of the original data, l τ (y i -γ0-x i T Bγ) is the Huber loss function, y i represents the i-th predicted data, γ0 is the intercept term of the estimated coefficient, x i represents the i-th original data, B represents the loading matrix, and γ represents the principal component regression coefficient.
4. The method according to claim 3, characterized in that The Huber loss function is expressed by the following formula: Among them, l τ (y i -γ0-x i T Bγ) represents the Huber loss function, and τ is the threshold parameter of the Huber loss function.
5. The method according to claim 1, characterized in that The second loss function is expressed by the following formula: Among them, L PCA (A, B) represents the second loss function, n represents the number of features of the original data, y i represents the i-th predicted data, X represents the data matrix composed of the original data, B represents the loading matrix, ω represents the weight coefficient, F represents the norm, and A represents the principal component matrix.
6. The method according to claim 1, characterized in that The first penalty function is expressed by the following formula: P1(B;λ B )=λ B ||B||1 Among them, P1(B;λ B ) represents the first penalty function, λ B represents the regularization parameter, and B represents the loading matrix.
7. The method according to claim 1, characterized in that The second penalty function is expressed by the following formula: Wherein, P2(γ,λ) represents the second penalty function, λ represents the regularization parameter, represents the SCAD penalty function, γ represents the principal component regression coefficient, and γ j is the jth principal component regression coefficient.
8. The method according to claim 7, characterized in that The SCAD penalty function is expressed by the following formula: in, represents the SCAD penalty function, I represents the indicative function, λ represents the tuning parameter, μ represents the input of the SCAD penalty function, and a represents the preset parameter.
9. A prediction device based on principal component regression, characterized in that: include: An acquisition module is used to acquire the original data to be predicted; A prediction module is used to use a target principal component regression model to predict the original data to obtain predicted data, wherein the target principal component regression model includes an objective function composed of a total loss function and a total penalty function, the total loss function includes a first loss function representing principal component regression and a second loss function used to represent principal component decomposition of the original data, and the total penalty function includes a first penalty function of principal component regression on the load matrix and a second penalty function of principal component regression on the regression coefficient.
10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the prediction method based on principal component regression as described in any one of claims 1 to 8.
11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program, when running, executes the prediction method based on principal component regression as described in any one of claims 1 to 8.
12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the prediction method based on principal component regression described in any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Petroleum exploration method and system based on artificial intelligence
CN120197148A
