Online unsupervised feature selection method for data streams based on contrastive learning
By comparing and learning online updates of prototypes and data distributions, a feature selection model is constructed, which solves the problem of insufficient feature discriminative power in online unsupervised feature selection methods, achieves high efficiency and resistance to forgetting, and improves clustering accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING FORESTRY UNIVERSITY
- Filing Date
- 2025-11-13
- Publication Date
- 2026-06-30
AI Technical Summary
Existing online unsupervised feature selection methods lack label information, resulting in weak feature discrimination and an inability to simultaneously guarantee adaptability, resistance to forgetting of learned knowledge, and efficiency.
A contrastive learning-based approach is adopted to construct a feature selection model by updating the prototype and data distribution online. The prototype probability is updated using k-means clustering and softmax operation, the data distribution parameters are calculated, a distribution contrastive loss function is constructed, and the feature selection matrix is solved by gradient descent to generate a feature selection index set.
It improves the discriminativeness and adaptability of feature selection, enables efficient adaptation to new data, effectively prevents the forgetting of learned knowledge, and improves clustering accuracy.
Smart Images
Figure CN121327552B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to an online unsupervised feature selection method for data streams based on contrastive learning. Background Technology
[0002] As data scale and complexity continue to grow, feature selection is becoming increasingly important in machine learning and data mining. Feature selection methods effectively address the "curse of dimensionality" problem. Based on whether label information is known, feature selection methods can be divided into supervised and unsupervised feature selection. However, in real-world applications, data label information is often unavailable. Furthermore, data is dynamic, arriving as data streams, which traditional supervised, offline feature selection methods cannot handle. Therefore, research on online unsupervised feature selection methods is receiving increasing attention.
[0003] There are relatively few existing online unsupervised feature selection methods. Due to the lack of label information, the selected features have weak discriminative power, which affects the classification results. In addition, they cannot simultaneously guarantee adaptability, the resistance to forgetting of learned knowledge, and efficiency. Summary of the Invention
[0004] In view of this, the present invention provides an online unsupervised feature selection method for data streams based on contrastive learning to solve the above problems.
[0005] This invention provides an online unsupervised feature selection method for data streams based on contrastive learning, comprising: acquiring a dataset to be processed and performing data preprocessing; uniformly dividing the preprocessed dataset into data blocks; using the data blocks as continuous input to update the prototype and data distribution online, and constructing an online feature selection model based on the updated prototype and updated data distribution; solving the online feature selection model to obtain the current feature selection matrix; after the data stream ends, obtaining the final feature selection matrix and generating an index set for feature selection; performing clustering based on the index set after feature selection and calculating the clustering accuracy.
[0006] In another implementation of the present invention, the step of using the data block as continuous input, updating the prototype and data distribution online, and constructing an online feature selection model based on the update results includes: using the data block as continuous input, clustering the initial time-phase data in the data block using k-means to obtain prototypes; calculating the cosine similarity between each sample and each prototype based on the prototype from the previous time-phase and the data in the currently newly received data block, and obtaining the probability that each sample belongs to each prototype through a softmax operation; updating the prototypes using the probability; calculating the initial parameters mean, covariance, and weights of the data distribution using the probability that each data point belongs to each prototype; updating the parameters of the data distribution at time t based on the parameters of the data distribution at time t-1, the data at time t, and the probability of each prototype; and constructing an online feature selection model based on the update results.
[0007] In another implementation of the present invention, the updated prototype is represented as:
[0008]
[0009] in, , representing the total number of samples received at time t and in the preceding time; This is a prototype based on the previous moment; This refers to the newly received data. The probability for each prototype.
[0010] In another implementation of the present invention, the updated data distribution is represented as follows:
[0011]
[0012] Where t represents time. It is the first The weights of a Gaussian distribution It is the first The probability density function of a Gaussian distribution. The mean, Let be the covariance matrix.
[0013] In another implementation of the present invention, the loss function of the online feature selection model is:
[0014]
[0015] in, and These are the distribution contrast loss and sparsity loss hyperparameters, This is the prototype loss.
[0016] In another implementation of the present invention, solving the online feature selection model to obtain the current feature selection matrix includes: solving the online feature selection model using the gradient descent method to obtain the current feature selection matrix.
[0017] In another implementation of the present invention, after the data stream ends, a final feature selection matrix is obtained, and an index set for feature selection is generated, including: after the data stream ends, a final feature selection matrix is obtained; based on the final feature selection matrix, the row norm of the matrix is calculated as the score of each feature, the k features with the highest scores are selected to form the dimensionality-reduced data, and an index set for feature selection is generated.
[0018] In another aspect, the present invention provides an online unsupervised feature selection system for data streams based on contrastive learning, comprising: a data preprocessing module for acquiring and preprocessing a dataset to be processed; uniformly dividing the preprocessed dataset into data blocks; a model building module for continuously updating the prototype and data distribution online using the data blocks as input, and constructing an online feature selection model based on the updated prototype and updated data distribution; a model solving module for solving the online feature selection model to obtain the current feature selection matrix; after the data stream ends, obtaining the final feature selection matrix and generating an index set for feature selection; and performing clustering based on the index set after feature selection to calculate the clustering accuracy.
[0019] In another aspect, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of an online data stream unsupervised feature selection method based on contrastive learning as described in any of the preceding claims.
[0020] In another aspect, the present invention provides a computer storage medium storing a computer program that, when executed by a processor, implements the steps of an online data stream unsupervised feature selection method based on contrastive learning as described in any of the preceding claims.
[0021] This invention presents an online unsupervised feature selection method for data streams based on contrastive learning. It utilizes online prototype updates and prototype contrastive learning to construct a loss contrastive function, thereby enhancing the discriminativeness of selected features and achieving efficient adaptability to new data. It also achieves cross-time data distribution estimation by updating the distribution online and proposes an efficient distribution contrastive learning method to construct a distribution contrastive loss function, thereby achieving efficient resistance to forgetting of learned knowledge. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. By reading the detailed description of the embodiments below, the advantages and benefits of the solutions will become clear to those skilled in the art. The accompanying drawings are only for illustrating preferred embodiments and are not intended to limit the present invention. In the accompanying drawings:
[0023] Figure 1 This is a schematic diagram of the process of an online data stream unsupervised feature selection method based on contrastive learning, according to an embodiment of the present invention. Detailed Implementation
[0024] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.
[0025] Figure 1 This is a flowchart illustrating an online unsupervised feature selection method based on contrastive learning, provided as an embodiment of the present invention. Figure 1 As shown, this embodiment mainly includes:
[0026] S101. Obtain the dataset to be processed and perform data preprocessing.
[0027] For example, data preprocessing includes mean removal and variance normalization.
[0028] S102. Divide the preprocessed dataset into data blocks evenly.
[0029] S103. Using the data block as continuous input, update the prototype and data distribution online, and construct an online feature selection model based on the updated prototype and updated data distribution.
[0030] S104. Solve the online feature selection model to obtain the current feature selection matrix.
[0031] S105. After the data stream ends, the final feature selection matrix is obtained, and the feature selection index set is generated.
[0032] S106. Clustering is performed based on the index set after feature selection, and the clustering accuracy is calculated.
[0033] This invention presents an online unsupervised feature selection method for data streams based on contrastive learning. It utilizes online prototype updates and prototype contrastive learning to construct a loss contrastive function, thereby enhancing the discriminativeness of selected features and achieving efficient adaptability to new data. It also achieves cross-time data distribution estimation by updating the distribution online and proposes an efficient distribution contrastive learning method to construct a distribution contrastive loss function, thereby achieving efficient resistance to forgetting of learned knowledge.
[0034] In another implementation of the present invention, the step of using the data block as continuous input, updating the prototype and data distribution online, and constructing an online feature selection model based on the update results includes: using the data block as continuous input, clustering the initial time-phase data in the data block using k-means to obtain prototypes; calculating the cosine similarity between each sample and each prototype based on the prototype from the previous time-phase and the data in the currently newly received data block, and obtaining the probability that each sample belongs to each prototype through a softmax operation; updating the prototypes using the probability; calculating the initial parameters mean, covariance, and weights of the data distribution using the probability that each data point belongs to each prototype; updating the parameters of the data distribution at time t based on the parameters of the data distribution at time t-1, the data at time t, and the probability of each prototype; and constructing an online feature selection model based on the update results.
[0035] In another implementation of the present invention, the updated prototype is represented as:
[0036]
[0037] in, , representing the total number of samples received at time t and in the preceding time; This is a prototype based on the previous moment; This refers to the newly received data. The probability for each prototype.
[0038] For example, for the initial moment, the prototype The initial time data can be analyzed using k-means. Obtained by clustering, Based on the prototype of the previous moment The newly received data is First, calculate each sample The cosine similarity with each prototype is calculated, and then a softmax operation is performed to obtain the sample. Probability of belonging to each prototype :
[0039]
[0040] in, This represents the dot product.
[0041] exist Based on this, calculate the updated probability-weighted prototype:
[0042]
[0043] in, This represents the total number of samples received at time t and in the preceding time.
[0044] In another implementation of the present invention, the updated data distribution is represented as follows:
[0045]
[0046] Where t represents time. It is the first The weights of a Gaussian distribution It is the first The probability density function of a Gaussian distribution. The mean, Let be the covariance matrix.
[0047] For example, at the initial moment, assume that the data follows a mixture Gaussian distribution with a probability density function as follows:
[0048]
[0049] in, It is the first The weights of a Gaussian distribution It is the first The probability density function of a Gaussian distribution, with mean covariance matrix Specifically:
[0050]
[0051] Using the probability that each data point belongs to each prototype Calculate the initial parameter mean of the data distribution covariance Weight initial mean Equivalent to the initial prototype ,for:
[0052]
[0053] The initial covariance is:
[0054]
[0055] The initial weights are:
[0056]
[0057] At time t, the parameters based on the data distribution at time t-1 and data at time t and the probability of each prototype Update the parameters for calculating the data distribution at time t. Specifically, this includes:
[0058] Update weights:
[0059]
[0060] Update the mean:
[0061]
[0062] Update covariance:
[0063]
[0064] Then the estimated global distribution at time t is:
[0065]
[0066] In another implementation of the present invention, the loss function of the online feature selection model is:
[0067]
[0068] in, and These are the distribution contrast loss and sparsity loss hyperparameters, This is the prototype loss.
[0069] For example, combining prototype contrastive loss, distribution contrastive loss, and sparse regularization forms the final loss function:
[0070]
[0071] in, and These are the distribution contrast loss and sparsity loss Hyperparameters, prototype loss as follows:
[0072]
[0073] in, express The prototype with the highest probability of belonging , Indicates the first Concentration estimates for each prototype family, A smaller value indicates a higher concentration, as defined below:
[0074]
[0075] in, It is a balancing parameter used to ensure that small families do not become too large. ,and It belongs to The number of samples.
[0076] Distribution contrast loss as follows:
[0077]
[0078] in, This is the overall mean.
[0079] Sparse loss as follows:
[0080]
[0081] In another implementation of the present invention, solving the online feature selection model to obtain the current feature selection matrix includes: solving the online feature selection model using the gradient descent method to obtain the current feature selection matrix. .
[0082] In another implementation of the present invention, after the data stream ends, a final feature selection matrix is obtained, and an index set for feature selection is generated, including: after the data stream ends, a final feature selection matrix is obtained; based on the final feature selection matrix, the row norm of the matrix is calculated as the score of each feature, the k features with the highest scores are selected to form the dimensionality-reduced data, and an index set for feature selection is generated.
[0083] For example, based on the final feature selection matrix Calculate the norm of the i-th row of the matrix. As a score for each feature, the k features with the highest scores are selected to form the dimensionality-reduced data. Based on the feature-selected dataset, k-means clustering is performed, and the clustering accuracy is calculated.
[0084] Example 1
[0085] Experimental verification was conducted using the method of this invention in a hardware environment consisting of an Intel(R) Core(TM) i5-13500H CPU 2.60GHz, 16GB RAM, and an NVIDIA RTX 3050 6GB Laptop GPU, and in a software environment consisting of Torch 2.7.0+cu128 and Torchvision0.22.0+cu128.
[0086] The experimental data used was the classic speech recognition dataset ISOLET, which has a total of 26 categories. The comparison methods are three online unsupervised feature selection methods. Method 1 is from the reference H. Huang, S. Yoo, SPKasiviswanathan, Unsupervised feature selection on data streams. In: Proceedings of the 24th ACM international conference on information and knowledge management. ACM, CIKM'15, 12015, 1031–1040; Method 2 is from the reference W. Shao, L. He, CT Lu, X. Wei X, PS Yu, Online unsupervised multi-view feature selection. In: Proceedings of the 16th IEEE international conference on data mining. IEEE, ICDM'16, 2016, 1203–1208; Method 3 is from the reference R. Ma, Y. Wang and L. Cheng, Feature selection on data stream via multi-clusterstructure preservation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020, 1065–1074.
[0087] Table 1. Clustering accuracy of four methods on ISOLET with 100 selected features.
[0088]
[0089] As can be seen from Table 1, the clustering accuracy obtained using the present invention is significantly higher than that of the other three online feature selection methods.
[0090] Example 2
[0091] The method of this invention can also be used for feature selection and unsupervised classification of the Tox-171 bioinformatics dataset. Each sample has a feature dimension of 5748, and these features are derived from microarray gene expression data. The samples include four classes.
[0092] Step 1: Preprocess all original features of the Tox-171 dataset by removing the mean and normalizing the variance, and then divide the dataset into 50 data blocks evenly.
[0093] Step 2: Continuously input data blocks, update the prototype and data distribution online, and build an online feature selection model.
[0094] Step 3: Solve for the current feature selection matrix by minimizing the loss function, including the following steps: optimization via gradient descent. get .
[0095] Step 4: After the data stream ends, obtain the final feature selection matrix, and then obtain the feature selection index set, including the following steps:
[0096] Based on the final feature selection matrix Calculate the norm of the i-th row of the matrix. As the score for each feature, the k features with the highest scores are selected to form the dimensionality-reduced data.
[0097] Step 5: Perform clustering based on the index set after feature selection, and calculate the clustering accuracy, including:
[0098] Based on the dataset after feature selection, the k-means method is used for clustering, and the clustering accuracy is calculated.
[0099] Table 2. Clustering accuracy of four methods on Tox-171 with 100 selected features.
[0100]
[0101] Therefore, the method of the present invention has a higher accuracy.
[0102] Another aspect of the present invention provides an online unsupervised feature selection system for data streams based on contrastive learning, comprising:
[0103] Data preprocessing module: acquires the dataset to be processed and performs data preprocessing; divides the preprocessed dataset into data blocks evenly.
[0104] Model building module: The data block is used as continuous input to update the prototype and data distribution online, and an online feature selection model is built based on the updated prototype and data distribution.
[0105] Model solving module: solves the online feature selection model to obtain the current feature selection matrix; after the data flow ends, the final feature selection matrix is obtained, and an index set of feature selection is generated; clustering is performed based on the index set after feature selection, and the clustering accuracy is calculated.
[0106] The present invention provides an online unsupervised feature selection system for data streams based on contrastive learning. It utilizes online prototype updates and prototype contrastive learning to construct a loss contrastive function, thereby enhancing the discriminativeness of selected features and achieving efficient adaptability to new data. It also achieves cross-time data distribution estimation by updating the distribution online and proposes an efficient distribution contrastive learning method to construct a distribution contrastive loss function, thereby achieving efficient resistance to forgetting of learned knowledge.
[0107] In another aspect of the present invention, the electronic device includes: a processor, a memory, and a communication bus and a communication interface.
[0108] in:
[0109] The processor, memory, and communication interface communicate with each other via a communication bus.
[0110] A communication interface is used to communicate with other electronic devices or servers.
[0111] The processor is used to execute programs, specifically, to perform any of the steps of the online data stream unsupervised feature selection method based on contrastive learning in the above embodiments.
[0112] Specifically, the program may include program code, which includes computer operation instructions.
[0113] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.
[0114] Memory is used to store programs. Memory may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device.
[0115] Specifically, the program can be used to cause the processor to execute the steps of any of the online data stream unsupervised feature selection methods based on contrastive learning described in the embodiments. The specific implementation of each step in the program can be found in the corresponding descriptions of the steps and units executed by any of the online data stream unsupervised feature selection methods based on contrastive learning described above, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments.
[0116] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of various embodiments of this application.
[0117] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.
[0118] Specific embodiments of the present invention have now been described. Other embodiments are within the scope of the appended claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result.
[0119] It should be noted that all directional indications (such as up, down, left, right, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship between the components in a certain order (as shown in the figure). If the specific order changes, the directional indication will also change accordingly.
[0120] In the description of this invention, the terms "first" and "second" are used only for convenience in describing different components or names, and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" and "second" may explicitly or implicitly include at least one of that feature.
[0121] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0122] It should be noted that although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of the present invention.
[0123] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be an improper limitation of the embodiments of the present invention.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An online data stream unsupervised feature selection method based on contrastive learning, characterized in that, include: Obtain the dataset to be processed and perform data preprocessing. The dataset to be processed is a speech recognition dataset or a bioinformatics dataset. The preprocessed dataset to be processed is evenly divided into data blocks; Using the data block as continuous input, the prototype and data distribution are updated online. An online feature selection model is constructed based on the updated prototype and data distribution, wherein the updated prototype is represented as: wherein, represents the number of all samples received at time t and in the past; is the prototype based on the previous time; is the current newly received data; is the probability of each prototype; The updated data distribution is represented as follows: where t denotes a time point, is the weight of the th Gaussian distribution, is the probability density function of the th Gaussian distribution, is the mean, is the covariance matrix. Solve the online feature selection model to obtain the current feature selection matrix; After the data stream ends, the final feature selection matrix is obtained, and an index set for feature selection is generated. Clustering is performed on the index set after feature selection, and the clustering accuracy is calculated.
2. The method according to claim 1, characterized in that, The step of using the data block as continuous input to update the prototype and data distribution online, and constructing an online feature selection model based on the update results, includes: Using the data block as continuous input, k-means is used to cluster the initial time data in the data block to obtain the prototype; Based on the prototype from the previous moment and the data in the newly received data block, calculate the cosine similarity between each sample and each prototype, and obtain the probability that each sample belongs to each prototype through a softmax operation. The prototype is updated using a weighted method based on the probabilities. The initial parameters of the data distribution, namely mean, covariance, and weights, are calculated using the probability that each data point belongs to each prototype. Based on the parameters of the data distribution at time t-1, the data at time t, and the probability of each prototype, update the parameters for calculating the data distribution at time t. An online feature selection model is constructed based on the updated results.
3. The method according to claim 1, characterized in that, The loss function of the online feature selection model is: in, and These are the distribution contrast loss and sparsity loss hyperparameters, This is the prototype loss.
4. The method according to claim 1, characterized in that, Solving the online feature selection model to obtain the current feature selection matrix includes: The online feature selection model is solved using the gradient descent method to obtain the current feature selection matrix.
5. The method according to claim 1, characterized in that, After the data stream ends, the final feature selection matrix is obtained, and an index set for feature selection is generated, including: After the data stream ends, the final feature selection matrix is obtained; Based on the final feature selection matrix, the row norm of the matrix is calculated as the score for each feature. The k features with the highest scores are selected to form the dimensionality-reduced data, and an index set for feature selection is generated.
6. An online unsupervised feature selection system for data streams based on contrastive learning, characterized in that, include: Data preprocessing module: acquires the dataset to be processed and performs data preprocessing, wherein the dataset to be processed is a speech recognition dataset or a bioinformatics dataset; The preprocessed dataset to be processed is evenly divided into data blocks; Model building module: This module uses the data blocks as continuous input to update the prototype and data distribution online. Based on the updated prototype and data distribution, it builds an online feature selection model, where the updated prototype is represented as: in, , representing the total number of samples received at time t and in the preceding time; This is a prototype based on the previous moment; This refers to the newly received data. The probability for each prototype; The updated data distribution is represented as follows: Where t represents time. It is the first The weights of a Gaussian distribution It is the first The probability density function of a Gaussian distribution. The mean, It is the covariance matrix; Model solving module: solves the online feature selection model to obtain the current feature selection matrix; after the data flow ends, the final feature selection matrix is obtained, and an index set of feature selection is generated; clustering is performed based on the index set after feature selection, and the clustering accuracy is calculated.
7. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the online data stream unsupervised feature selection method based on contrastive learning as described in any one of claims 1 to 5.
8. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of the online data stream unsupervised feature selection method based on contrastive learning as described in any one of claims 1 to 5.