Small sample regression method fusing LightGBM and TabNet adaptive feature processing
By combining the LightGBM and TabNet models, high-order features are generated and adaptively pruned, which solves the problems of insufficient feature relationship processing and generalization ability in small sample regression tasks, and achieves efficient and stable prediction results.
Patent Information
- Application Number
- CN202511010586.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-22
- Publication Date
- 2025-10-31
AI Technical Summary
Existing machine learning models struggle to handle complex feature relationships and maintain generalization ability in small-sample regression tasks, making them prone to overfitting. Furthermore, the presence of redundant features leads to decreased model stability and prediction accuracy.
By combining the LightGBM and TabNet models, high-order features are generated through TabNet and iteratively processed. LightGBM is used to quantify feature contribution and perform pruning. An adaptive feature generation and pruning method (ARFGP) is constructed to dynamically identify and remove redundant features and optimize the feature space.
It significantly improves the prediction accuracy and stability of small sample regression tasks, reduces model complexity, and enhances generalization ability. It is suitable for fields where data is scarce but prediction accuracy is required, such as medical diagnosis and business economic forecasting.
Smart Images
Figure CN120874018A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, specifically to a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing. Background Technology
[0002] In today's data-driven era, despite the rapid development of big data technologies and machine learning methods, practical applications, especially in fields such as medical diagnosis, remote sensing analysis, genomics, and corporate economic forecasting, often face challenges such as high data acquisition costs and scarce labeled samples. In these situations, traditional machine learning models perform well when handling large-scale datasets. For example, LightGBM (Light Gradient Boosting Machine) has been able to effectively handle large-scale datasets and has achieved significant results in regression tasks. However, in small-sample regression tasks, these models often face challenges such as limited training samples, complex feature relationships, and insufficient generalization ability. Alternatively, TabNet (Tabular Neural Network), as a deep learning model, excels at capturing complex feature interactions in high-dimensional data, but it is prone to overfitting in small-sample regression tasks and has high computational resource requirements. In other words, for small-sample regression tasks, existing traditional machine learning models struggle to simultaneously handle complex feature relationships and maintain generalization ability, easily leading to overfitting.
[0003] In recent years, many studies have attempted to combine machine learning and deep learning methods to compensate for the shortcomings of single methods. However, these methods are still insufficient in optimizing feature selection and redundant feature removal. Especially when dealing with high-dimensional data, the presence of redundant features may lead to a decrease in model stability and prediction accuracy, particularly when the data is sparse or noisy. Summary of the Invention
[0004] To address the technical problem that existing machine learning models cannot simultaneously handle feature relationships and maintain generalization ability in few-shot regression tasks, leading to overfitting, and that redundant features cause a decrease in model stability and prediction accuracy, this invention aims to provide a few-shot regression method that integrates LightGBM and TabNet adaptive feature processing. The specific technical solution adopted is as follows:
[0005] Obtain the UCI regression dataset and get the original feature matrix and original target variable of the UCI regression dataset;
[0006] The TabNet model is used to expand and replace the original feature matrix and the original target variables in sequence, and the total feature matrix is established through iterative processing.
[0007] The contribution of each feature in the total feature matrix is quantitatively evaluated using the LightGBM model. The total feature matrix is then pruned based on the contribution. The pruned total feature matrix is analyzed, and its dimensions are adjusted to obtain the retained feature matrix.
[0008] A regression model is constructed by integrating the TabNet and LightGBM models. The retained feature matrix is then input into the regression model for prediction, and the prediction results are output.
[0009] Preferably, the original feature matrix and the original target variables are sequentially expanded and replaced using the TabNet model, and the total feature matrix is established through iterative processing, including:
[0010] The TabNet model is used to perform regression analysis on the original feature matrix and the original target variable to generate predicted values. The predicted values are defined as new features and are then concatenated with the original feature matrix to form an expanded feature matrix.
[0011] Calculate the correlation between the expanded feature matrix and the original target variable, and rank the importance of the features based on the correlation.
[0012] The feature with the highest importance ranking is selected to replace the original target variable, generating an updated target variable;
[0013] The updated target variable is used as the new input to the TabNet model. The TabNet model is rerun and iterated until all features in the original feature matrix are replaced, forming the total feature matrix.
[0014] Preferably, the predicted value is generated, and the corresponding calculation formula is:
[0015]
[0016] in, Indicates the predicted value. This indicates the regression operation of the TabNet model; Represents the original feature matrix;
[0017] The predicted values are defined as new features, and are concatenated with the original feature matrix to form an expanded feature matrix. The corresponding calculation formula is as follows:
[0018]
[0019] in, This represents the expanded feature matrix.
[0020] Preferably, the correlation between the expanded feature matrix and the original target variable is calculated, and a ranking of feature importance is established based on the correlation, including:
[0021] The correlation between any feature in the expanded feature matrix and the original target variable is calculated using the following formula:
[0022]
[0023] in, Represents the th element in the expanded characteristic matrix. The correlation between each feature and the original target variable; Represents the th element in the expanded characteristic matrix. The first sample Features, original feature matrix , Indicates the size of the original feature matrix. Indicates the number of features; Represents the th element in the expanded characteristic matrix. The mean of each feature; Indicates the first The original target variables for each sample; This represents the mean of the original target variable;
[0024] Similarly, determine the correlation between all features in the expanded feature matrix and the original target variable, and establish a ranking of feature importance. The corresponding calculation formula is as follows:
[0025]
[0026] in, Indicates the first Ranking of the importance of each feature; This represents the index of the ranked features.
[0027] Preferably, the contribution of each feature in the total feature matrix is quantitatively evaluated using the LightGBM model. Based on the contribution, the total feature matrix is pruned. The pruned total feature matrix is analyzed, and its dimensions are adjusted to obtain the retained feature matrix, including:
[0028] The gain of each feature in the total feature matrix is calculated using the LightGBM model to quantify the corresponding contribution and establish a contribution ranking.
[0029] Pruning conditions are set based on contribution, which are used to prune the total feature matrix and obtain the pruned total feature matrix.
[0030] Based on the pruned feature matrix, if more than half of the features in the total feature matrix fail to enter the top 50% of contribution ranking, the pruned total feature matrix is defined as the target input. The target input is combined with the TabNet model to generate a new expanded feature matrix, which is then concatenated with the original feature matrix to form a new input matrix.
[0031] The input matrix is re-evaluated using the LightGBM model to assess the contribution of each feature and then pruned. The pruned input matrix is iteratively processed until all features in the total feature matrix are in the top 50% of contribution or the total feature matrix has only one feature left. The iteration is then stopped, and the retained feature matrix is obtained.
[0032] Preferably, the gain of each feature in the total feature matrix is calculated using the LightGBM model, and the corresponding calculation formula is as follows:
[0033]
[0034] in, Indicates the first One feature in all The sum of cumulative gains across the trees; Indicates the total number of trees; Indicates the first The feature in the first The change in gain within a tree.
[0035] Preferably, pruning conditions are set based on contribution to prune the total feature matrix and obtain the pruned total feature matrix. The corresponding calculation formula is as follows:
[0036]
[0037] in, This represents the total feature matrix after pruning; This indicates a pruning operation; Represents the total characteristic matrix; Represents the first element in the total characteristic matrix. One feature; Indicates the first The feature in the first Gain in a tree; This represents the threshold, i.e., the lower limit of the contribution.
[0038] To address the aforementioned issues, this application also provides an electronic device comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus, and the processor invokes logical instructions stored in the memory to execute a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing as described in any of the preceding claims.
[0039] The present invention has the following beneficial effects:
[0040] 1. By combining the high-order feature generation capabilities of the TabNet model with the overfitting suppression mechanism of the LightGBM model, and introducing adaptive recursive feature generation and pruning methods, this approach effectively addresses the accuracy and robustness challenges faced by small sample datasets in regression tasks. It enables the extraction of deeper feature representations from limited data, and through iterative optimization, ensures that the model maintains strong prediction accuracy and stability even under complex feature interactions and extreme value conditions. This is of great significance for fields such as medical diagnosis and financial risk assessment where data is scarce but prediction accuracy is extremely important. Specifically, the regression method proposed in this application is defined as ARFGP (Adaptive Recursive Feature Generation). Adaptive Recursive Feature Generation and Pruning (ARPP) dynamically generates high-order feature matrices rich in information and intelligently identifies and removes redundant or less contributing features through a recursive pruning mechanism. This dynamic feature space optimization significantly improves model training efficiency, reduces model complexity, and enhances prediction accuracy and generalization ability. It also significantly reduces reliance on large-scale datasets, making it particularly suitable for data-constrained applications, especially in situations of data scarcity or imbalance. This allows for seamless integration into other few-shot learning tasks and applicability to various practical fields, such as business economic forecasting, medical diagnosis, remote sensing analysis, and genomics. Through its flexible feature generation and pruning mechanism, this method adapts to diverse data characteristics and task requirements, demonstrating strong generalization ability and stability, and providing a general and efficient framework for solving various few-shot regression problems.
[0041] 2. By combining the TabNet and LightGBM models, we can not only accurately capture the complex nonlinear interactions between features, overcoming the limitations of traditional tree models in this regard, but also enable the model to more comprehensively understand the potential patterns contained in the data, improving its ability to handle high-dimensional and nonlinear features. Furthermore, based on the LightGBM model, we provide a more intuitive evaluation of feature contribution, enhancing the interpretability of the regression model in practical applications. This allows users to understand why the regression model makes specific predictions, thereby improving the transparency and reliability of the model in complex business scenarios.
[0042] 3. The electronic device provided by this invention has the same beneficial effects as the few-sample regression method that integrates LightGBM and TabNet adaptive feature processing provided by this invention, and will not be described in detail here. Attached Figure Description
[0043] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 The overall framework diagram of a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing provided in an embodiment of the present invention;
[0045] Figure 2 This invention provides a few-shot regression method that integrates LightGBM and TabNet adaptive feature processing based on the Concrete dataset, where Figure (a) represents LightGBM; Figure (b) represents TabNet; and Figure (c) represents ARFGP.
[0046] Figure 3 This invention provides a few-shot regression method that integrates LightGBM and TabNet adaptive feature processing based on the California_housing dataset, where Figure (a) represents LightGBM; Figure (b) represents TabNet; and Figure (c) represents ARFGP.
[0047] Figure 4 This is an example of a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing, based on the Concrete dataset, and provides an error analysis diagram of multiple regression prediction algorithms. In this diagram, (a) represents LightGBM; (b) represents TabNet; and (c) represents ARFGP.
[0048] Figure 5 This is an example of a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing, based on the California_housing dataset. The error analysis diagram shows the regression prediction of multiple algorithms. In this example, Figure (a) represents LightGBM; Figure (b) represents TabNet; and Figure (c) represents ARFGP.
[0049] Figure 6 A comparison of prediction errors of three algorithms—LightGBM, TabNet, and ARFGP—based on the Concrete dataset, provided in an embodiment of the present invention, for a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing.
[0050] Figure 7 A comparison of prediction errors of three algorithms—LightGBM, TabNet, and ARFGP—based on the California_housing dataset, provided in an embodiment of the present invention, is shown.
[0051] Figure 8 This is a comparison of the computational costs of three algorithms—LightGBM, TabNet, and ARFGP—based on the Concrete dataset and the California_housing dataset, respectively, in a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing, according to an embodiment of the present invention. Detailed Implementation
[0052] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a small-sample regression method integrating LightGBM and TabNet adaptive feature processing proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0054] The following description, in conjunction with the accompanying drawings, details the specific scheme of a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing provided by this invention.
[0055] Please see Figure 1 This diagram illustrates the overall framework of a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing, according to an embodiment of the present invention. The method includes:
[0056] Step S1: Obtain the UCI regression dataset and get the original feature matrix and original target variable of the UCI regression dataset;
[0057] Step S2: The original feature matrix and the original target variables are expanded and replaced sequentially using the TabNet model, and the total feature matrix is established through iterative processing;
[0058] Step S3: Quantitatively evaluate the contribution of each feature in the total feature matrix using the LightGBM model, prune the total feature matrix according to the contribution, analyze the pruned total feature matrix, and adjust it according to the dimension of the total feature matrix to obtain the retained feature matrix.
[0059] Step S4: Integrate the TabNet model and the LightGBM model to construct a regression model, input the retained feature matrix into the regression model for prediction, and output the prediction results.
[0060] To better illustrate this, in scenarios with scarce data, i.e., small sample sizes, traditional regression models struggle to effectively capture deep patterns in the data, are susceptible to noise and outliers, and suffer significant reductions in prediction accuracy and robustness. Due to limited data volume, models are prone to overlearning noise and random patterns in the training data, leading to poor performance on unseen new data. Furthermore, real-world data often contains highly nonlinear and complex feature interactions, affecting the model's ability to capture complex feature relationships. In addition, high-dimensional data often contains a large number of redundant or irrelevant features, which not only increases the computational burden but may also introduce noise, reducing the model's predictive performance. Moreover, existing models suffer from poor interpretability; their "black box" nature limits their application in scenarios requiring high transparency. That is, the model's internal working mechanisms and decision-making processes are opaque to external users, making it difficult for them to understand the model's decision-making process and thus affecting their overall judgment.
[0061] Therefore, this application proposes a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing. This method combines the advantages of the traditional LightGBM tree model and the TabNet deep learning model. It generates high-order features through TabNet and effectively integrates them using the robustness of LightGBM to more comprehensively capture complex patterns in the data. This improves prediction performance while focusing on model interpretability. Furthermore, by introducing effective regularization and feature selection mechanisms, it reduces model complexity, mitigates the risk of overfitting, and enhances the model's generalization ability. Finally, through adaptive feature generation and pruning mechanisms, it dynamically identifies and removes redundant features, optimizes the feature space, and improves the model's efficiency and accuracy.
[0062] It can be noted that the few-shot regression method proposed in this application, which integrates LightGBM and TabNet adaptive feature processing, is defined as an adaptive recursive feature generation and pruning (ARFGP) method combining LightGBM (Light Gradient Boosting Machine) and TabNet (Tabular Neural Network). It combines the end-to-end feature generation capability of TabNet with the robustness of LightGBM. Through a recursive feature generation and pruning process, it dynamically adjusts the importance and selection strategy of features to build a multi-stage, adaptive feature processing and model integration framework. This framework can adapt to different data characteristics, achieve efficient feature modeling and selection, and significantly improve the accuracy and robustness of few-shot regression tasks.
[0063] Preferably, in this embodiment, the UCI regression dataset consists of the Concrete Compressive Strength dataset and the California Housing dataset. The Concrete Compressive Strength dataset contains the components of concrete, such as cement, blast furnace slag, fly ash, water, plasticizer, coarse aggregate, and fine aggregate, as well as the age and corresponding concrete compressive strength. It is used to predict concrete strength, and the Y-label value distribution typically exhibits some skewness, but is generally relatively concentrated. The California Housing dataset contains housing characteristics of various regions in California, such as median income, house age, average number of rooms, average number of bedrooms, population, average occupancy rate, latitude and longitude, and the corresponding median house price. The Y-label value distribution typically exhibits a long-tail effect, meaning that most areas have lower house prices, while a few areas have higher house prices. The Y-label value distribution refers to the distribution of the actual observed values of the target variable (Y) throughout the entire dataset in the regression analysis, reflecting the statistical characteristics and potential patterns of the data.
[0064] Understandably, the TabNet model processes input features through a self-attention mechanism, generates higher-order features, and performs feature interactions to improve the model's ability to handle complex data.
[0065] Further, step S2 includes:
[0066] Step S21: Perform regression analysis on the original feature matrix and the original target variable using the TabNet model to generate predicted values. Define the predicted values as new features and concatenate them with the original feature matrix to form an expanded feature matrix.
[0067] The explanation is as follows: taking the original feature matrix and the original target variable as input parameters, TabNet processes the input parameters through its self-attention mechanism. Feature interaction is completed through multiple "feature selection" layers and "feature generation" layers. That is, TabNet generates feature masks through self-attention mechanism and weighted scaling factor, dynamically selects relevant features corresponding to each input parameter, and enhances the expressive power of features. The use of feature masks helps the TabNet model identify key features and effectively reduce noise, thereby improving the model's prediction accuracy.
[0068] Further, in step S21, the predicted value is generated, and the corresponding calculation formula is:
[0069]
[0070] in, Indicates the predicted value. This indicates the regression operation of the TabNet model; Represents the original feature matrix;
[0071] The predicted values are defined as new features, and are concatenated with the original feature matrix to form an expanded feature matrix. The corresponding calculation formula is as follows:
[0072]
[0073] in, This represents the expanded feature matrix.
[0074] Specifically, the original feature matrix Compared with the original target variable Input the data into the TabNet model for regression analysis, where... This represents the total number of feature samples in the original feature matrix; Indicates the size of the original feature matrix; outputs the predicted value. This is used to enhance feature interaction capabilities; then, the predicted values are concatenated with the original feature matrix as new features to form an expanded feature matrix, denoted as... .
[0075] Step S22: Calculate the correlation between the expanded feature matrix and the original target variable, and establish a ranking of feature importance based on the correlation.
[0076] It can be explained that the Pearson correlation coefficient is used to measure the correlation between each feature in the expanded feature matrix and the original target variable. In other words, by calculating the Pearson correlation coefficient, the strength of the linear relationship between the two parameters is quantified, and its value range is... -1 indicates a perfect negative correlation, 0 indicates no correlation, and 1 indicates a perfect positive correlation. This helps to filter data based on conditions and optimize the feature selection process.
[0077] Further, step S22 includes:
[0078] Step S221: Calculate the correlation between any feature in the expanded feature matrix and the original target variable. The corresponding calculation formula is:
[0079]
[0080] in, Represents the th element in the expanded characteristic matrix. The correlation between each feature and the original target variable; Represents the th element in the expanded characteristic matrix. The first sample Features, original feature matrix , Indicates the size of the original feature matrix. Indicates the number of features; Represents the th element in the expanded characteristic matrix. The mean of each feature; Indicates the first The original target variables for each sample; This represents the mean of the original target variable;
[0081] Step S222: Similarly, determine the correlation between all features in the expanded feature matrix and the original target variable, and establish the importance ranking of the features. The corresponding calculation formula is as follows:
[0082]
[0083] in, Indicates the first Ranking of the importance of each feature; This represents the index of the ranked features.
[0084] Specifically, after obtaining the correlation between each feature and the original target variable through the Pearson correlation coefficient, the features are ranked according to the magnitude of the correlation to establish a ranking of feature importance. The feature with the highest correlation to the original target variable is selected to help reduce irrelevant features and reduce the complexity of the model.
[0085] Step S23: Select the feature with the highest importance ranking to replace the original target variable, generating an updated target variable; that is, based on the importance ranking, obtain the feature with the highest relevance, and record the predicted value corresponding to this feature as... Replace the original target variable Generate the updated target variable, denoted as ,Right now .
[0086] Step S24: Use the updated target variable as the new input to the TabNet model, rerun the TabNet model, and continue iterating until all features in the original feature matrix have been replaced to form the total feature matrix.
[0087] Specifically, the updated target variable As new input to the TabNet model, the TabNet model is rerun to generate the next round of predicted values. These predicted values are then used as new features and concatenated with the original feature matrix to continue expansion, resulting in an expanded feature matrix. This process is iteratively repeated until all features in the original feature matrix have been replaced, achieving full coverage of the predicted features generated by the TabNet model. In other words, the original target variable is replaced by the most relevant features, and a new feature matrix is recursively generated to continuously optimize feature selection and generate higher-order features, thereby improving the model's ability to process complex data.
[0088] Understandably, LightGBM is used to suppress overfitting by introducing regularization terms or performing pruning, thereby enhancing the model's generalization ability. It maintains high accuracy while rapidly processing datasets and ultimately generates prediction results with high stability.
[0089] Furthermore, step S3 includes:
[0090] Step S31: Calculate the gain of each feature in the total feature matrix using the LightGBM model to quantify the corresponding contribution and establish a contribution ranking.
[0091] Understandably, LightGBM evaluates the contribution of each feature during training through its gradient boosting tree algorithm, typically by calculating the gain of a feature to assess its importance, i.e., to maximize information gain or reduce error.
[0092] Further, in step S31, the gain of each feature in the total feature matrix is calculated using the LightGBM model, and the corresponding calculation formula is as follows:
[0093]
[0094] in, Indicates the first One feature in all The sum of cumulative gains across the trees; Indicates the total number of trees; Indicates the first The feature in the first The change in gain within a tree.
[0095] Specifically, based on the relationships between trees, i.e., the sum of the gain changes after each iteration, the gain of each feature is calculated, and its contribution is quantified. Then, the features are ranked in descending order according to the magnitude of the gain to establish a contribution ranking. That is, if any feature in the total feature matrix generated by the TabNet model does not show a significant gain in the LightGBM model, ARFGP will determine that the feature has a small contribution to the model and proceed with subsequent operations.
[0096] Step S32: Set pruning conditions based on contribution to prune the total feature matrix and obtain the pruned total feature matrix; that is, set the pruning conditions of ARFGP according to the contribution obtained in step S31, i.e., when the gain is greater than a set threshold. At the same time, retain the corresponding features and remove features that contribute little to the model, reduce redundant information, reduce model complexity and improve training efficiency.
[0097] Further, in step S32, pruning conditions are set based on contribution to prune the total feature matrix, and the pruned total feature matrix is obtained. The corresponding calculation formula is as follows:
[0098]
[0099] in, This represents the total feature matrix after pruning; This indicates a pruning operation; Represents the total characteristic matrix; Represents the first element in the total characteristic matrix. One feature; Indicates the first The feature in the first Gain in a tree; This represents the threshold, i.e., the lower limit of the contribution.
[0100] Step S33: Based on the pruned feature matrix, if more than half of the features in the total feature matrix fail to rank in the top 50% of contribution, the pruned total feature matrix is defined as the target input. The target input is then combined with the TabNet model to generate a new expanded feature matrix, which is then concatenated with the original feature matrix to form a new input matrix.
[0101] To clarify, when analyzing the contribution of features, if more than half of the features retained after pruning fail to rank in the top 50% of contribution, the amount of data at this point affects the subsequent analysis process. Therefore, the feature matrix is dimensionally adjusted to remove some features retained during pruning. Specifically, assuming the total feature matrix has a dimensionality of [missing value], the pruned feature matrix will have a dimension of [missing value]. After adjustment, the dimension of the feature matrix was reduced to... After removing redundant features, the reduced feature matrix is used as input to generate a new expanded feature matrix through the TabNet model, denoted as . Then the feature matrix With the original feature matrix The matrices are concatenated to form a new input matrix, denoted as . .
[0102] Step S34: Re-evaluate the contribution of each feature in the input matrix using the LightGBM model and prune it. Iterate through the pruned input matrix until all features in the total feature matrix are in the top 50% of contribution or the total feature matrix has only one feature left. Stop iterating and obtain the retained feature matrix.
[0103] Specifically, the input matrix obtained in step S33 The input is fed into the LightGBM model to re-evaluate the contribution of each feature, establish a new contribution ranking, and perform pruning. Then, based on the pruned feature matrix, a judgment is made according to step S33 to determine whether the condition is met. If not, steps S33-S34 are continued for iterative processing, i.e., the dimension of the pruned feature matrix is adjusted. Gradually reduced to , ... and after each iteration, ARFGP dynamically adjusts the dimension and composition of the feature matrix based on the contribution output of the LightGBM model until all features in the total feature matrix are in the top 50% of the contribution ranking or the total feature matrix has only one feature left. Then the iteration stops, and the retained feature matrix is obtained, which retains the features most representative of the regression task.
[0104] It can be explained that in step S4, the TabNet model and the LightGBM model are integrated to construct a regression model. The retained feature matrix is input into the regression model for prediction, and the prediction result is output. Specifically, the TabNet model and the LightGBM model processed in the previous steps are integrated to establish a regression model. That is, through the synergistic effect of the two, the overall prediction accuracy and generalization ability of the model are effectively improved. The model can significantly capture the complex interaction relationship between features and achieve an efficient and adaptive feature selection and generation process. Among them, the TabNet model, through its unique attention mechanism, can select the most informative features layer by layer to obtain high-order features, thereby improving the interpretability and accuracy of the model. The LightGBM model, with its efficient gradient boosting framework, can quickly process the dataset and significantly reduce the consumption of computing resources while maintaining high prediction accuracy.
[0105] Understandably, by combining the high-order feature generation capabilities of the TabNet model with the overfitting suppression mechanism of the LightGBM model, and introducing adaptive recursive feature generation and pruning methods, the accuracy and robustness challenges faced by small sample datasets in regression tasks are effectively addressed. This allows for the extraction of deeper feature representations from limited data, and through iterative optimization, ensures that the model maintains strong prediction accuracy and stability even under complex feature interactions and extreme value conditions. This is of great significance for fields such as medical diagnosis and financial risk assessment where data is scarce but prediction accuracy is extremely important. Specifically, the regression method proposed in this application is defined as ARFGP (Adaptive Recursive Feature Generation and Pruning). Pruning (Adaptive Recursive Feature Generation and Pruning) dynamically generates high-order feature matrices rich in information and intelligently identifies and removes redundant or less contributing features through a recursive pruning mechanism. This dynamic feature space optimization significantly improves model training efficiency, reduces model complexity, and enhances prediction accuracy and generalization ability. It also significantly reduces reliance on large-scale datasets, making it particularly suitable for data-constrained applications, especially in situations of data scarcity or imbalance. This allows for seamless integration into other few-shot learning tasks and applicability to various practical fields, such as business economic forecasting, medical diagnosis, remote sensing analysis, and genomics. Through its flexible feature generation and pruning mechanism, this method adapts to diverse data characteristics and task requirements, demonstrating strong generalization ability and stability, and providing a general and efficient framework for solving various few-shot regression problems.
[0106] By combining the TabNet and LightGBM models, we can not only accurately capture the complex nonlinear interactions between features, overcoming the limitations of traditional tree models in this regard, but also enable the model to more comprehensively understand the potential patterns contained in the data, improving its ability to handle high-dimensional and nonlinear features. Furthermore, based on the LightGBM model, we provide a more intuitive evaluation of feature contribution, enhancing the interpretability of the regression model in practical applications. This allows users to understand why the regression model makes specific predictions, thereby improving the transparency and reliability of the model in complex business scenarios.
[0107] To better illustrate and verify the effectiveness of ARFGP in small-sample regression tasks, experiments were conducted on two UCI regression datasets, Concrete Compressive Strength and California Housing, used in this embodiment, to evaluate performance and determine the effectiveness and advantages of ARFGP.
[0108] The ARFGP proposed in this application was compared with five existing algorithms based on two UCI regression datasets, Concrete Compressive Strength and California Housing. The regression performance metrics were analyzed: MSE (Mean Squared Error), representing the average of the squared differences between predicted and actual values; RMSE (Root Mean Squared Error), representing the standard deviation between predicted and actual values; MAE (Mean Absolute Error), representing the average of the absolute differences between predicted and actual values; R², representing the coefficient of determination, indicating the proportion of the dependent variable variation explained by the model; and MAPE (Mean Absolute Percentage Error), representing the average of the absolute percentage differences between predicted and actual values. Table 1 shows the experimental results comparing the accuracy of different algorithms in regression tasks.
[0109] Table 1. Experimental results comparing the accuracy of different algorithms in regression tasks.
[0110] It can be observed that ARFGP outperforms other comparative models in all performance metrics on the Concrete Compressive Strength and California Housing datasets. Specifically, on the Concrete Compressive Strength dataset, ARFGP achieves the lowest MSE (15.0272), RMSE (3.8765), and MAE (2.6006), as well as the highest R² (0.9417) and the lowest MAPE (8.33%), reflecting that ARFGP has the highest prediction accuracy and best fit on this dataset. Compared to the LightGBM model alone, which has an MSE of 18.0140, ARFGP's performance is significantly improved, validating its effectiveness in small-sample regression tasks. The TabNet model performs similarly to Catboost on this dataset, but not as well as the LightGBM model, which may be related to its tendency to overfit in small-sample scenarios.
[0111] On the California Housing dataset, ARFGP also performed exceptionally well, achieving the lowest MSE (0.1878) and MAE (0.2799), as well as a relatively high R² (0.8567) and a low MAPE (15.79%). Notably, only the RMSE of 0.4334 was slightly higher than the LightGBM model's 0.4325, but overall performance remained superior. The TabNet model performed relatively poorly on this dataset, with its MSE (0.4073) and MAPE (22.61%) both higher than other models, reflecting the challenges that the TabNet model may face when dealing with small sample data with long-tailed distributions and extreme values. ARFGP, by combining the robustness of the LightGBM model, effectively mitigated this disadvantage of the TabNet model.
[0112] Please combine Figure 2 and Figure 3 The feature contribution maps of different algorithms are compared. The Y-axis represents the features sorted by global contribution, the X-axis represents the distribution of SHAP (SHapley Additive exPlanations) values, and different colors represent the magnitude of the feature values, from blue to red.
[0113] To illustrate, in the Concrete dataset, features such as "Age" and "Cement" show high SHAP values, indicating that these features have a significant impact on the prediction of concrete compressive strength. Compared with other models, ARFGP is more reasonable in its feature contribution ranking. The gains of higher-order features generated by the TabNet model in the LightGBM model are high when used to evaluate contribution, indicating that ARFGP successfully integrates higher-order features from the TabNet model, improving the model's predictive ability. In the California_housing dataset, features such as "MedInc" and "AveOccup" have high SHAP values, reflecting the importance of these features in predicting house prices. This demonstrates that, compared to other comparative models, ARFGP can better capture the nonlinear relationships between features, improving the model's interpretability and prediction accuracy.
[0114] Please combine Figure 4 and Figure 5 The regression prediction error analysis charts of different algorithms are compared, where the X-axis represents the actual value and the Y-axis represents the predicted value.
[0115] To clarify, a comparison of the two datasets shows that both LightGBM and TabNet demonstrate some ability in fitting the actual and predicted values, but their scatter distribution is relatively dispersed, especially in extreme value regions, resulting in larger prediction errors. This indicates that these two models still have limitations in capturing complex interactions between features and accurately modeling the target variable. In contrast, the ARFGP method outperforms other algorithms; its scatter distribution is denser, with most points distributed within the ideal range. The proximity of the ideal line indicates that the ARFGP method has smaller errors and higher fitting accuracy when predicting actual values.
[0116] Please combine Figure 6 and Figure 7 It compares the prediction errors of different algorithms on the Concrete dataset and the California_housing dataset, where the X-axis represents the prediction error and the Y-axis represents the density.
[0117] To clarify, in the Concrete dataset, the peak of ARFGP's error distribution density is concentrated in the low error range, i.e., the range of 0-1. However, its error distribution tail is slightly longer than other models such as LightGBM, indicating that the prediction bias is slightly larger on a few samples. In addition, ARFGP's KDE (Kernel Density Estimate) curve is generally skewed to the left, and the density in the low error region is relatively high, indicating that its predictions are more accurate on most samples. In contrast, LightGBM's error distribution tail is shorter, but the proportion of high-error samples is slightly higher, especially in the error range of 3-5, where there is still a large distribution density, indicating that the prediction error is large on some samples. TabNet's error distribution is the most dispersed, with a significant tail extension, and the KDE curve is flat, indicating that its ability to handle high-error samples is insufficient.
[0118] In the California Housing dataset, ARFGP exhibits a shorter tail in its error distribution, with density peaks concentrated in the low error range (0-1), demonstrating stronger prediction accuracy and stability. In contrast, LightGBM's error distribution has a slightly longer tail, still accounting for a high proportion in the 1-1.5 error range, indicating larger errors for some samples. TabNet's error distribution remains relatively dispersed, with a long right tail extension, suggesting poor performance on complex samples.
[0119] Therefore, ARFGP shows better low-error coverage in the Concrete dataset; in the CaliforniaHousing dataset, ARFGP shows better generalization ability and accuracy; in contrast, LightGBM and TabNet are insufficient in handling high-error samples; while ARFGP, by utilizing adaptive feature generation and pruning mechanisms, shows higher robustness and prediction accuracy in small-sample regression tasks.
[0120] Please see Figure 8 The graph shows a comparison of the computational costs of three algorithms, LightGBM, TabNet, and ARFGP, based on the Concrete dataset and the California_housing dataset, respectively. The X-axis represents the dataset, and the Y-axis represents the computation time cost (seconds).
[0121] It should be noted that the ARFGP method consumes more computational resources compared to the standalone LightGBM and TabNet models. TabNet, in particular, requires significant computational resources through its self-attention mechanism and end-to-end feature selection, especially in high-dimensional feature processing, consuming more memory and computation time. Furthermore, although LightGBM's histogram-based splitting algorithm optimizes memory usage and computational efficiency, the ARFGP method's feature generation and pruning process involves multiple iterative steps and requires importance evaluation of a large number of features, increasing the overall computational burden. To improve computational efficiency, the experiments employed the Optuna framework for hyperparameter optimization and rationally allocated and scheduled computational resources, thereby minimizing computational overhead while ensuring prediction accuracy. Optionally, in practical applications, sacrificing some computational time and resources to improve prediction accuracy is understandable and acceptable.
[0122] In summary, ARFGP demonstrates superior performance on two datasets with different characteristics, especially in few-shot regression tasks, where its prediction accuracy and generalization ability outperform models using LightGBM or TabNet alone, as well as other mainstream ensemble learning algorithms; fully demonstrating the effectiveness of the adaptive recursive feature generation and pruning method.
[0123] The second embodiment of the present invention proposes an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus, and the processor calls logical instructions in the memory to execute a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing as described in any of the foregoing embodiments; this electronic device has the same beneficial effects as the aforementioned few-sample regression method that integrates LightGBM and TabNet adaptive feature processing, and will not be described in detail here.
[0124] Understandably, when a module of an electronic device is in operation, it needs to utilize the small sample regression method that integrates LightGBM and TabNet adaptive feature processing provided in the foregoing embodiments. Therefore, whether the method and program data are integrated or different hardware is configured to produce a function with similar effect to that achieved by the present invention, it falls within the protection scope of the present invention.
[0125] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0126] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
Claims
1. A few-sample regression method integrating LightGBM and TabNet adaptive feature processing, characterized in that, The method includes: Obtain the UCI regression dataset and get the original feature matrix and original target variable of the UCI regression dataset; The TabNet model is used to expand and replace the original feature matrix and the original target variables in sequence, and the total feature matrix is established through iterative processing. The contribution of each feature in the total feature matrix is quantitatively evaluated using the LightGBM model. The total feature matrix is then pruned based on the contribution. The pruned total feature matrix is analyzed, and its dimensions are adjusted to obtain the retained feature matrix. A regression model is constructed by integrating the TabNet and LightGBM models. The retained feature matrix is then input into the regression model for prediction, and the prediction results are output.
2. The few-sample regression method integrating LightGBM and TabNet adaptive feature processing according to claim 1, characterized in that, The TabNet model is used to sequentially expand and replace the original feature matrix and the original target variables, and then iteratively processes them to build the total feature matrix, including: The TabNet model is used to perform regression analysis on the original feature matrix and the original target variable to generate predicted values. The predicted values are defined as new features and are then concatenated with the original feature matrix to form an expanded feature matrix. Calculate the correlation between the expanded feature matrix and the original target variable, and rank the importance of the features based on the correlation. The feature with the highest importance ranking is selected to replace the original target variable, generating an updated target variable; The updated target variable is used as the new input to the TabNet model. The TabNet model is rerun and iterated until all features in the original feature matrix are replaced, forming the total feature matrix.
3. The few-sample regression method integrating LightGBM and TabNet adaptive feature processing according to claim 2, characterized in that, The formula for generating predicted values is as follows: ； in, Indicates the predicted value. This indicates the regression operation of the TabNet model; Represents the original feature matrix; The predicted values are defined as new features, and are concatenated with the original feature matrix to form an expanded feature matrix. The corresponding calculation formula is as follows: ； in, This represents the expanded feature matrix.
4. The few-sample regression method integrating LightGBM and TabNet adaptive feature processing according to claim 2, characterized in that, Calculate the correlation between the expanded feature matrix and the original target variable, and rank the importance of features based on the correlation, including: The correlation between any feature in the expanded feature matrix and the original target variable is calculated using the following formula: ； in, Represents the th element in the expanded characteristic matrix. The correlation between each feature and the original target variable; Represents the th element in the expanded characteristic matrix. The first sample Features, original feature matrix , This represents the size of the original feature matrix. Indicates the number of features; Represents the th element in the expanded characteristic matrix. The mean of each feature; Indicates the first The original target variables for each sample; This represents the mean of the original target variable; Similarly, determine the correlation between all features in the expanded feature matrix and the original target variable, and establish a ranking of feature importance. The corresponding calculation formula is as follows: ； in, Indicates the first Ranking of the importance of each feature; This represents the index of the ranked features.
5. The few-sample regression method integrating LightGBM and TabNet adaptive feature processing according to claim 1, characterized in that, The contribution of each feature in the total feature matrix is quantitatively evaluated using the LightGBM model. Based on the contribution, the total feature matrix is pruned. The pruned feature matrix is analyzed, and its dimensions are adjusted to obtain the retained feature matrix, including: The gain of each feature in the total feature matrix is calculated using the LightGBM model to quantify the corresponding contribution and establish a contribution ranking. Pruning conditions are set based on contribution, which are used to prune the total feature matrix and obtain the pruned total feature matrix. Based on the pruned feature matrix, if more than half of the features in the total feature matrix fail to enter the top 50% of contribution ranking, the pruned total feature matrix is defined as the target input. The target input is combined with the TabNet model to generate a new expanded feature matrix, which is then concatenated with the original feature matrix to form a new input matrix. The input matrix is re-evaluated using the LightGBM model to assess the contribution of each feature and then pruned. The pruned input matrix is iteratively processed until all features in the total feature matrix are in the top 50% of contribution or the total feature matrix has only one feature left. The iteration is then stopped, and the retained feature matrix is obtained.
6. The few-sample regression method integrating LightGBM and TabNet adaptive feature processing according to claim 5, characterized in that, The gain of each feature in the total feature matrix is calculated using the LightGBM model, and the corresponding calculation formula is as follows: ； in, Indicates the first One feature in all The sum of cumulative gains across the trees; Indicates the total number of trees; Indicates the first The feature in the first The change in gain within a tree.
7. The few-sample regression method integrating LightGBM and TabNet adaptive feature processing according to claim 5, characterized in that, The pruning conditions are set based on contribution, which are used to prune the total feature matrix and obtain the pruned total feature matrix. The corresponding calculation formula is as follows: ； in, This represents the total feature matrix after pruning; This indicates a pruning operation; Represents the total characteristic matrix; Represents the first element in the total characteristic matrix. One feature; Indicates the first The feature in the first Gain in a tree; This represents the threshold, i.e., the lower limit of the contribution.
8. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor calls logical instructions from the memory to execute a few-sample regression method that integrates LightGBM and TabNet adaptive feature processing as described in any one of claims 1 to 7.