Method for exploring feature selection in streaming data scene by using large language model
By combining the large language model and ToT thinking tree technology, features in streaming data scenarios are dynamically selected, which solves the problems of insufficient feature selection accuracy and real-time performance of existing methods in streaming data scenarios, realizes intelligent and adaptive update of feature selection, and improves the prediction accuracy and efficiency of the model.
Patent Information
- Application Number
- CN202510915592.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-03
- Publication Date
- 2025-10-17
AI Technical Summary
Existing feature selection methods have difficulty capturing data changes in real time in streaming data scenarios, lack semantic understanding and dynamic updating capabilities, resulting in insufficient feature selection accuracy and real-time performance, and inability to effectively utilize the intelligent decision-making capabilities of large language models.
A large language model combined with ToT thinking tree technology is used to extract initial features, initialize data drift sliding window parameters, use Wasserstein distance to measure data drift, and use the GPT model to evaluate feature importance. Feature subsets are dynamically selected for incremental training to form the final feature set.
It significantly improves the accuracy and real-time performance of feature selection, enhances the model's generalization ability and scene adaptability, realizes the deep utilization of semantic information, and optimizes the model training process.
Smart Images
Figure CN120804643A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of feature selection, and in particular to a method for exploring feature selection in a streaming data scenario using a large language model. Background Art
[0002] In the field of streaming data processing and modeling, feature selection is a core component that influences model performance and training efficiency. Streaming data is continuously and dynamically generated. In specific experimental scenarios, only a subset of samples from the global dataset are initially available, with the remaining samples gradually observed over time. Furthermore, the data distribution can exhibit concept drift or semantic evolution. In this scenario, feature selection must balance dynamic adaptability, semantic understanding, and real-time updating capabilities, but existing methods generally have limitations.
[0003] Current mainstream feature selection methods include filter-based methods, embedded methods, online learning and incremental methods, and methods based on expert rules or fixed templates. While filter-based methods are computationally efficient, they are unable to perceive the temporal continuity and drift characteristics of streaming data; embedded methods struggle to cope with data semantics or distribution drift, resulting in poor feature selection accuracy; online learning and incremental methods experience delayed feature selection responses when faced with dynamic changes in high-dimensional features or drastic data drift; and expert rule-based methods lack versatility and intelligent decision-making capabilities. These methods share significant common flaws: they struggle to ensure feature validity in environments with data drift and semantic evolution, lack dynamic update capabilities, underutilize semantic information, and fail to integrate with emerging technologies such as large language models.
[0004] Therefore, there is an urgent need for a new feature selection framework that can capture data changes in real time, fuse semantic information, combine emerging technologies, and support online incremental updates to break through the performance bottleneck of existing methods. Summary of the Invention
[0005] In response to the above-mentioned problems in the prior art, the present invention provides a method for exploring feature selection in streaming data scenarios using a large language model, which effectively improves the accuracy and real-time performance of feature selection and realizes the in-depth utilization of semantic information.
[0006] To achieve the above objectives, the present invention proposes a method for exploring feature selection in streaming data scenarios using a large language model, comprising:
[0007] S1. Extract initial features: Summarize the background information and feature-related information of the dataset into prompt words, and use the summarized prompt words in combination with the ToT thinking tree technology to guide the large language model to select initial features;
[0008] S2. Initialize the data drift sliding window parameter settings: Initialize the downstream model and historical record container; the historical record container includes accuracy, drift, and feature importance;
[0009] S3, stream data input and preprocessing: receive a new batch of stream data every round, extract features and labels, update the data window and use the Wasserstein distance to measure the current data drift;
[0010] S4, obtain feature importance construction prompt words and batch request feature importance: obtain feature importance construction prompt words evaluated by GPT large model, and batch request important features of current data to GPT model;
[0011] S5, analyze GPT results and count global feature scores: convert GPT output into feature scores, sort feature scores, and select Top-N proportion of features as dynamically selected feature subset;
[0012] S6, model training and updating: combine the dynamically selected feature subset with the initial feature set to form the final feature set, use the final feature set for prediction and incremental training, calculate the current accuracy and record it, update the accuracy, drift and feature importance history record container, and use the updated accuracy, drift and feature importance history record container as auxiliary reference to input the next round of GPT prompt;
[0013] S7, repeat S3-S6 until the stream data experiment ends.
[0014] Preferably, in S1, the specific process of selecting initial features by the large language model is to explore different branches of ToT thinking tree through three steps of generation, evaluation and selection, and gradually approach the optimal solution.
[0015] Preferably, in S2, the initialization data drift sliding window parameters include the window size of the data drift sliding window, the initial parameters of the downstream online learning models FTRL, XGB and LSVM, and the size of the accuracy, drift and feature importance history record container.
[0016] Preferably, in S3, the expression of Wasserstein distance is:
[0017] W p (P,Q)=(inf r∈γ(P,Q) E (x,y)～r ||x-y|| p ) 1 / p ;
[0018] In the formula, γ(P,Q) represents all joint distribution sets of P and Q, and ||x-y|| is the distance measure of samples x and y.
[0019] In the Wasserstein distance expression, when p is equal to 1, the Wasserstein-1 distance is used to measure the current data drift, and the minimum cost required to convert the distribution P to Q is calculated. The formula can be expressed as follows in the discrete case:
[0020]
[0021] In the formula, r ij represents the minimum transmission cost from the sample x i to y j , and satisfies j r ij = P(x i ) and i r ij = Q(y j ).
[0022] Preferably, in S3, the drift score is obtained by separately calculating the Wasserstein-1 distance for each feature dimension and then taking the average value, and the calculation formula is as follows:
[0023]
[0024] In the formula, d is the feature dimension, P k and Q k are the distributions of the kth feature in the current batch and the historical window, respectively.
[0025] Preferably, in S4, the prompt constructed includes information of the feature importance of the previous round, the drift score, the model accuracy, and requires the large language model GPT to return the important feature list of each sample.
[0026] Preferably, in S5, the function function is constructed to analyze the GPT result and count the global feature score, the importance score of each feature is calculated by using the function function, and the sorting selection is given.
[0027] Preferably, in S6, the final feature set is used for prediction and incremental training, which is performed in the downstream online model learning models FTRL, XGB and LSVM.
[0028] Therefore, the present application proposes a method for exploring feature selection in a stream data scenario using a large language model, which has the following beneficial effects:
[0029] (1) Improve the accuracy and real-time performance of feature selection: The present application combines ToT thinking tree and large language model semantic understanding, dynamically perceives data distribution changes, and continuously optimizes feature selection strategy, significantly improving the accuracy and real-time response capability of feature selection.
[0030] (2) Deeply excavate semantic association and adaptability: the application analyzes the semantic association between features by a large language model, and uses incremental learning and sliding window mechanism, so that the feature selection strategy can be self-adaptive to the dynamic evolution of stream data, and the generalization and scene adaptability of the model are enhanced.
[0031] The technical solutions of the application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS
[0032] Figure 1 is a size comparison chart of the selected data set of the method for exploring feature selection in the stream data scene by using a large language model;
[0033] Figure 2 is an accuracy comparison chart of each algorithm of the method for exploring feature selection in the stream data scene by using a large language model in the downstream model LSVM model and FTRL model, wherein the proportion of selected features is fixed at 20%;
[0034] Figure 3 is an accuracy comparison chart of each algorithm of the method for exploring feature selection in the stream data scene by using a large language model in the downstream model XGBoost model, wherein the proportion of selected features is fixed at 20%;
[0035] Figure 4 is an accuracy comparison chart of DFSM-LLM and full feature of the method for exploring feature selection in the stream data scene by using a large language model;
[0036] Figure 5 is an accuracy comparison chart of DFSM-LLM before and after removing ToT initialization of the method for exploring feature selection in the stream data scene by using a large language model;
[0037] Figure 6 is an accuracy comparison chart of DFSM-LLM before and after removing the dynamic selection framework of the method for exploring feature selection in the stream data scene by using a large language model. DETAILED DESCRIPTION
[0038] In order to make the technical solutions, advantages and purposes of the application clearer, the technical solutions of the embodiments of the application will be described clearly and completely below. The described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the described embodiments of the application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the protection scope of the present application.
[0039] Unless otherwise defined, the technical terms or scientific terms used in the application shall have the usual meanings understood by those skilled in the art.
[0040] The application provides a method for exploring feature selection in a stream data scenario by using a large language model, comprising:
[0041] S1, extracting initial features: summarizing background information and feature-related information of a data set into prompt words, and using the summarized prompt words in combination with ToT thinking tree technology to guide the large language model to select initial features;
[0042] ToT thinking tree technology is one of the prompt word engineering technologies in the large language model, and the core idea is to decompose a problem into multiple intermediate states, and each state represents a stage or idea for solving the problem. The specific process of the large language model selecting initial features is to explore on different branches of the ToT thinking tree through three steps of generation, evaluation and selection, and gradually approach the optimal solution.
[0043] S2, initialization of data drift sliding window parameter setting: initialization of a downstream model and a history record container; wherein the history record container includes accuracy, drift and feature importance;
[0044] Among them, the initialization of the data drift sliding window parameter includes the window size of the data drift sliding window, the initial parameters of the downstream online learning model FTRL, XGB and LSVM, and the size of the accuracy, drift and feature importance history record container.
[0045] S3, stream data input and preprocessing: receiving a new batch of stream data every round, extracting features and labels, updating the data window and using the Wasserstein distance to measure the current data drift;
[0046] The expression of the Wasserstein distance is:
[0047] W p (P,Q)=(inf r∈γ(P,Q) E (x,y)～r ||x-y|| p ) 1 / p ;
[0048] In the formula, γ(P,Q) represents the set of all joint distributions of P and Q, and ||x-y|| is the distance measure of samples x and y;
[0049] Among them, in the expression of the Wasserstein distance, when p is equal to 1, the Wasserstein-1 distance is used to measure the current data drift, and the minimum cost required to convert distribution P to Q, the calculation formula can be expressed as:
[0050]
[0051] In the formula, r ijdenotes the minimum transportation cost from sample x i to y j , and satisfies ∑ j r ij = P(x i ) and ∑ i r ij = Q(y j ).
[0052] The drift score is obtained by separately calculating the Wasserstein-1 distance for each feature dimension and then taking the average, and the calculation formula is as follows:
[0053]
[0054] In the formula, d is the feature dimension, P k and Q k are the distributions of the kth feature in the current batch and the historical window, respectively.
[0055] S4, obtaining feature importance, constructing prompt words, and batch requesting feature importance: obtaining the feature importance constructed prompt word evaluated by the GPT large model, and batch requesting the important features of the current data to the GPT model;
[0056] Among them, the constructed prompt words include the information of the feature importance of the last round, the drift score, the model accuracy and the requirement of the large language model GPT to return the important feature list of each sample.
[0057] S5, analyzing GPT results and counting global feature scores: converting GPT output into feature scores, sorting feature scores, and selecting Top-N proportion of features as a dynamically selected feature subset;
[0058] In S5, the function function is constructed to analyze the GPT results and count the global feature scores, and the importance score of each feature is calculated by using the function function, and the sorting selection is given.
[0059] S6, model training and updating: combining the dynamically selected feature subset with the initial feature set to form a final feature set, using the final feature set for prediction and incremental training, calculating the current accuracy and recording, updating the accuracy, drift, and feature importance history record container, and using the updated accuracy, drift, and feature importance history record container as an auxiliary reference to input the next round of GPT prompts;
[0060] In S6, using the final feature set for prediction and incremental training is performed in the downstream online model learning models FTRL, XGB and LSVM.
[0061] S7, repeating S3-S6 until the streaming data experiment ends.
[0062] Embodiment one,
[0063] As Figures 1-6 shown in Table 1, seven public datasets were used to evaluate DFSM-LLM in Example One, all of which were from the UCI website and Kaggle website. All the datasets used are described in Table 1:
[0064] Table 1 Description of experimental datasets
[0065]
[0066]
[0067] Table 2 and Table 3 give the experimental results of the accuracy of all comparison methods on the datasets. The ranking of each comparison method on the dataset is recorded in the brackets, and if several methods have the same ranking result, the same ranking is recorded. The last row shows the average (ranking) result in each table. From Table 2 and Table 3, it can be observed that the method proposed in the present application is superior to all other comparison methods in terms of average accuracy.
[0068] Table 2 Experimental results using lsvm and frtl models for downstream model
[0069] Dataset DFSM-LLM lap_score mcfs max_variance offesel spambase 0.7689(1) 0.6875(4) 0.6602(5) 0.7270(3) 0.7546(2) Diabetes 0.7265(1) 0.7224(3) 0.721(4) 0.7239(2) 0.7059(5) Dota2 0.5104(3) 0.5022(4) 0.5010(5) 0.5188(1) 0.5133(2) NPHA 0.4873(5) 0.4982(4) 0.5386(1) 0.5111(2) 0.5111(2) pc1 0.9292(1) 0.8858(4) 0.9158(2) 0.8816(5) 0.9017(3) KDD99 0.9534(1) 0.8380(5) 0.9230(2) 0.8888(3) 0.8590(4) german 0.6370(1) 0.5600(4) 0.5730(3) 0.6230(2) 0.5580(5) Average rank 0.7161(1) 0.6706(5) 0.6904(3) 0.6963(2) 0.6862(4)
[0070] Table 3 Experimental results using xgb model for downstream model
[0071]
[0072]
[0073] Table 4, Table 5 and Table 6 give the experimental results of the accuracy of all ablation experiments on the datasets. The ranking of each comparison on the dataset is recorded in the brackets.
[0074] Table 4 Ablation experiment results, comparison of accuracy using DFSM-LLM method and without using the method
[0075] Dataset DFSM-LLM full spambase 0.7689(1) 0.6753(2) Diabetes 0.7265(1) 0.7025(2) Dota2 0.5104(1) 0.5075(2) NPHA 0.4873(1) 0.3120(2) pc1 0.9292(1) 0.8710(2) KDD99 0.9534(1) 0.8750(2) German 0.6370(1) 0.5288(2)
[0076] Table 5 Ablation experiment results, comparison of accuracy before and after removing the ToT framework in DFSM-LLM method
[0077]
[0078]
[0079] Table 6 Ablation experiment results, comparison of accuracy before and after removing the dynamic feature selection framework in DFSM-LLM method
[0080]
[0081] In terms of ablation experiments, the experimental results of the full feature space and the feature space selected by the method in this paper show that in the 7 data sets, the model accuracy is higher than that of the full feature space after using DFSM-LLM method to select features. This shows that this method can effectively filter out key features, remove redundant information, avoid the problem of model overfitting or increased computational complexity caused by too many features, and thus improve the generalization ability and prediction accuracy of the model.
[0082] For the ablation experiment of the ToT framework module, after removing the framework, the accuracy of the algorithm on multiple data sets decreased significantly. In the spambase data set, the accuracy decreased from 0.7689 to 0.7153; in the Diabetes data set, the accuracy decreased from 0.7265 to 0.6874; in the KDD99 data set, the accuracy decreased from 0.9534 to 0.8412.
[0083] Although the accuracy on the Dota2 data set increased slightly, from the overall data set performance, the ToT framework has important significance for improving the performance of the algorithm. However, overall, the ToT framework can help the model better initialize and optimize the model training process, making the model more efficient and accurate in feature learning and classification decision-making, and is one of the key components that ensures the good results of DFSM-LLM method.
[0084] For the ablation experiment of the dynamic selection framework module, after removing the framework, the accuracy of the algorithm on multiple data sets also decreased significantly, with a larger decrease. In the spambase data set, the accuracy decreased from 0.7689 to 0.6814; in the Diabetes data set, the accuracy decreased from 0.7265 to 0.578; in the pc1 data set, the accuracy decreased from 0.9534 to 0.8214.
[0085] From the overall data set performance, the dynamic selection framework module is the core component of the algorithm. It can help the model better adapt to the streaming data scenario and optimize the model training process. This core framework is the most important component that ensures the good results of DFSM-LLM method.
[0086] Therefore, the application provides a method for exploring feature selection in a streaming data scenario by using a large language model, which significantly improves the intelligent level and dynamic adaptation ability of feature selection. The semantic-driven screening mechanism based on the ToT thinking tree can deeply analyze the potential correlation between features, capture data distribution drift in real time by combining the Wasserstein distance, and realize adaptive updating of the feature set through a dynamic sliding window, ensuring that the model always matches the latest data pattern; the feature selection strategy has self-evolution ability. Compared with traditional methods, this method effectively improves the prediction accuracy, reduces feature redundancy, effectively improves the accuracy and real-time performance of feature selection, and also realizes the deep utilization of semantic information, providing an innovative solution for efficient modeling and accurate prediction in the streaming data scenario.
[0087] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand: the technical solutions of the present application can still be modified or replaced by the equivalent, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.
Claims
1. A method for exploring feature selection in streaming data scenarios using a large language model, characterized in that: include: S1. Extract initial features: Summarize the background information and feature-related information of the dataset into prompt words, and use the summarized prompt words in combination with the ToT thinking tree technology to guide the large language model to select initial features; S2. Initialize the data drift sliding window parameter settings: Initialize the downstream model and historical record container; the historical record container includes accuracy, drift, and feature importance; S3, streaming data input and preprocessing: Each round receives a new batch of streaming data, extracts features and labels, updates the data window, and uses Wasserstein distance to measure the current data drift; S4. Obtain feature importance constructing prompt words and batch request feature importance: Obtain feature importance constructing prompt words evaluated by the GPT large model, and batch request important features of the current data from the GPT model; S5. Analyze GPT results and calculate global feature scores: Convert GPT output into feature scores, sort the feature scores, and select the top-N features as the dynamically selected feature subset. S6. Model training and updating: The dynamically selected feature subset is combined with the initial feature set to form the final feature set. The final feature set is used for prediction and incremental training. The current accuracy is calculated and recorded. The accuracy, drift, and feature importance history record container is updated. The updated accuracy, drift, and feature importance history record container is used as an auxiliary reference to input the next round of GPT prompts. S7. Repeat S3-S6 until the streaming data experiment ends.
2. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 1, characterized in that: In S1, the specific process of the large language model selecting the initial features is to explore different branches of the ToT thinking tree through three steps: generation, evaluation, and selection, and gradually approach the optimal solution.
3. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 1, characterized in that: In S2, the initialized data drift sliding window parameters include the window size of the data drift sliding window, the initial parameters of the downstream online learning models FTRL, XGB, and LSVM, and the size of the accuracy, drift, and feature importance history record container.
4. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 1, characterized in that: In S3, the expression of Wasserstein distance is: W p (P,Q)=(inf r∈γ(P,Q) E (x,y)～r ||x-y|| p ) 1 / p ; Where γ(P,Q) represents the set of all joint distributions of P and Q, ||xy|| is the distance measure between samples x and y; In the Wasserstein distance expression, when p is equal to 1, the Wasserstein-1 distance is used to measure the current data drift and the minimum cost required to convert the distribution P to Q. The calculation formula in the discrete case can be expressed as: Where r ij Indicates that from the sample x i to y j The minimum transmission cost, and satisfy ∑ j r ij =P(x i ) and ∑ i r ij =Q(y j ).
5. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 4, characterized in that: In S3, the drift score is obtained by calculating the Wasserstein-1 distance for each feature dimension separately and then taking the average value. The calculation formula is: Where d is the feature dimension, P k and Q k are the distributions of the kth feature in the current batch and the historical window, respectively.
6. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 1, characterized in that: In S4, the constructed prompt words include information about the importance of features in the previous round, drift scores, model accuracy, and require the large language model GPT to return a list of important features for each sample.
7. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 1, characterized in that: In S5, a function is constructed to parse the GPT results and calculate the global feature scores. The function is used to calculate the importance score of each feature and give a ranking selection.
8. The method for exploring feature selection in streaming data scenarios using a large language model according to claim 1, characterized in that: In S6, the final feature set is used for prediction and incremental training in the downstream online model learning models FTRL, XGB, and LSVM.