Method, device, equipment and computer storage medium for obtaining high-dimensional features of data
By preprocessing and one-hot encoding the raw data, and combining it with machine learning algorithms to generate high-dimensional features, the problems of information loss and masking of important features in existing technologies are solved, achieving feature acquisition with high accuracy and wide applicability.
Patent Information
- Application Number
- CN202011182603.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-10-29
AI Technical Summary
In existing technologies, feature engineering suffers from problems such as information loss, high computational resource consumption, and insufficient model accuracy during machine learning. In particular, when generating cross-term features, important features may be masked by unimportant features, making it unsuitable for deep learning algorithms.
By preprocessing the original data, the first feature is generated using one-hot encoding, and the second feature is generated by combining it with a machine learning algorithm. After merging, the importance is judged, and the feature is copied and expanded when the preset threshold is met, thus generating a high-dimensional feature.
It improves the accuracy and discriminative power of features, is applicable to machine learning and deep learning algorithms, and enhances the classification accuracy of the model.
Smart Images

Figure CN114429166B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method, device, electronic device and computer storage medium for acquiring high-dimensional features of data. Background Art
[0002] Machine learning is a common method for exploring data. In the process of machine learning, feature engineering is to convert data into features that can better represent potential problems, extract more information from the original features to the maximum extent for machine learning, and further improve the performance of machine learning algorithms. In the prior art, the PCA algorithm is used to achieve dimensionality reduction after the features are one-hot encoded. In the process of performing one-hot encoding, the features can also be cross-featured; the features are screened by performing collinearity selection and feature importance selection, and then the model is constructed. The PCA algorithm often has a certain degree of ambiguity in its interpretation of its meaning, which is not as complete as the original sample. The principal components with a small contribution rate may often contain important distinguishing information about sample differences, which will affect the overall classification results. In addition, new synthetic features are generated by adding, subtracting, multiplying and dividing two or more features. Such features may not necessarily reflect the implicit laws between features. When a large number of features are selected and used in machine learning algorithms, important features may be covered by other unimportant features.
[0003] Using the solutions in the prior art, artificial experience is used to segmentally encode numerical features or perform PCA dimensionality reduction on features, which may cause some information to be lost. When generating cross-term features, most of them use enumeration methods, which consume a lot of computing resources for calculation, are time-consuming, and may generate many invalid features; directly building a model for the filtered features, in most machine learning, other non-important features may cover up important features, resulting in inaccurate model classification; the full amount of features generated can only be used for machine learning algorithms, and cannot reach a certain level suitable for deep learning algorithms.
[0004] Therefore, how to obtain high-dimensional features with high accuracy and significant feature distinction is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0005] The embodiments of the present application provide a method, device, electronic device and computer storage medium for obtaining high-dimensional features of data, which can obtain high-dimensional features with high accuracy and significant feature distinction.
[0006] In a first aspect, an embodiment of the present application provides a method for obtaining high-dimensional features of data, comprising:
[0007] Get the original data;
[0008] Preprocess the original data to obtain processed data;
[0009] Perform one-hot encoding on the processed data to obtain the first feature;
[0010] Generate a second feature using the processed data based on a machine learning algorithm;
[0011] Merge the first feature and the second feature, and determine whether the importance of the merged feature meets a preset importance threshold;
[0012] When it is determined that the preset importance threshold is met, the merged features are copied, expanded, and merged to obtain high-dimensional features.
[0013] Optionally, the original data is preprocessed to obtain processed data, including:
[0014] At least one of deleting, filling the mode and cleaning the original data are performed on the outliers and missing values to obtain processed data.
[0015] Optionally, the processed data is one-hot encoded to obtain the first feature, including:
[0016] At least one attribute register is used to encode each attribute of the processed data to obtain a first feature.
[0017] Optionally, merging the first feature and the second feature, and determining whether the importance of the merged feature meets a preset importance threshold, includes:
[0018] The first feature and the second feature are combined, and the importance of the combined feature is determined by using a random forest algorithm;
[0019] Determine whether the importance of each merged feature meets the preset importance threshold.
[0020] In a second aspect, an embodiment of the present application provides a device for acquiring high-dimensional features of data, including:
[0021] An acquisition module is used to obtain raw data;
[0022] A preprocessing module is used to preprocess the original data to obtain processed data;
[0023] An encoding module, used for performing one-hot encoding on the processed data to obtain a first feature;
[0024] A generating module, for generating a second feature using the processed data based on a machine learning algorithm;
[0025] A merging module, used to merge the first feature and the second feature, and determine whether the importance of the merged feature meets a preset importance threshold;
[0026] The copy and expansion module is used to copy and expand the merged features to obtain high-dimensional features when it is determined that the preset importance threshold is met.
[0027] Optional, pre-processing modules, including:
[0028] The preprocessing unit is used to delete, fill in the mode and clean the original data to obtain processed data.
[0029] Optional, encoding module, including:
[0030] The encoding unit is used to encode each attribute of the processed data using at least one attribute register to obtain a first feature.
[0031] Optionally, merge modules including:
[0032] A merging unit, used to merge the first feature and the second feature, and determine the importance of the merged feature using a random forest algorithm;
[0033] The judging unit is used to judge whether the importance of each merged feature meets a preset importance threshold.
[0034] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;
[0035] When the processor executes the computer program instructions, the method for obtaining high-dimensional features of data as shown in the first aspect is implemented.
[0036] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method for acquiring high-dimensional features of data as shown in the first aspect is implemented.
[0037] The high-dimensional feature acquisition method, device, electronic device and computer storage medium of the data of the embodiment of the present application can acquire high-dimensional features with high accuracy and significant feature distinction. The high-dimensional feature acquisition method of the data includes: acquiring original data; preprocessing the original data to obtain processed data; performing one-hot encoding on the processed data to obtain a first feature; generating a second feature using the processed data based on a machine learning algorithm; merging the first feature and the second feature, and determining whether the importance of the merged feature meets a preset importance threshold; if it is determined that the preset importance threshold is met, copying and expanding the merged feature to obtain a high-dimensional feature with high accuracy and significant feature distinction. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0039] Figure 1 It is a flowchart of a method for obtaining high-dimensional features of data provided by an embodiment of the present application;
[0040] Figure 2 is a flow chart of a method for obtaining high-dimensional features of data provided by another embodiment of the present application;
[0041] Figure 3 It is a structural schematic diagram of a device for acquiring high-dimensional features of data provided by an embodiment of the present application;
[0042] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0043] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0044] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0045] Using the solutions in the prior art, artificial experience is used to segmentally encode numerical features or perform PCA dimensionality reduction on features, which may cause some information to be lost. When generating cross-term features, most of them use enumeration methods, which consume a lot of computing resources for calculation, are time-consuming, and may generate many invalid features; directly building a model for the filtered features, in most machine learning, other non-important features may cover up important features, resulting in inaccurate model classification; the full amount of features generated can only be used for machine learning algorithms, and cannot reach a certain level suitable for deep learning algorithms.
[0046] In order to solve the problems of the prior art, the embodiments of the present application provide a high-dimensional feature acquisition method, device, electronic device and computer storage medium. The following first introduces the high-dimensional feature acquisition method of data provided by the embodiments of the present application.
[0047] Figure 1 FIG. 1 is a flow chart showing a method for obtaining high-dimensional features of data provided by an embodiment of the present application. Figure 1 As shown, the method for obtaining high-dimensional features of the data includes:
[0048] S101, obtaining original data;
[0049] S102, preprocessing the original data to obtain processed data;
[0050] In one embodiment, the raw data is preprocessed to obtain processed data, including:
[0051] At least one of deleting, filling the mode and cleaning the original data are performed on the outliers and missing values to obtain processed data.
[0052] S103, performing one-hot encoding on the processed data to obtain a first feature;
[0053] In one embodiment, performing one-hot encoding on the processed data to obtain the first feature includes: using at least one attribute register to encode each attribute of the processed data to obtain the first feature.
[0054] S104, generating a second feature using the processed data based on a machine learning algorithm;
[0055] S105, merging the first feature and the second feature, and determining whether the importance of the merged feature meets a preset importance threshold;
[0056] In one embodiment, the first feature and the second feature are merged, and it is determined whether the importance of the merged feature meets a preset importance threshold, including: merging the first feature and the second feature, and using a random forest algorithm to determine the importance of the merged feature; and determining whether the importance of each merged feature meets a preset importance threshold.
[0057] S106: When it is determined that the preset importance threshold is met, the merged features are copied, expanded, and merged to obtain high-dimensional features.
[0058] The method for obtaining high-dimensional features of the data includes: obtaining original data; preprocessing the original data to obtain processed data; performing one-hot encoding on the processed data to obtain a first feature; generating a second feature based on a machine learning algorithm using the processed data; merging the first feature and the second feature, and judging whether the importance of the merged feature meets a preset importance threshold; and when it is determined that the preset importance threshold is met, copying and expanding the merged feature to obtain a high-dimensional feature with high accuracy and significant feature distinction.
[0059] The above technical solution is described below with a specific embodiment.
[0060] The process of the method for obtaining high-dimensional features of data provided by another embodiment of the present application can be found in Figure 2 The specific implementation is as follows:
[0061] One-hot encoding feature module:
[0062] 1) Non-numeric variable processing:
[0063] Delete outliers and missing values and fill the majority to remove invalid data, then use one-hot encoding to encode non-numeric variables, and use i-bit attribute registers to encode i attributes of the feature. Each attribute will become an independent feature, so some variables will all be expanded into dummy variables.
[0064] 2) Numerical variable processing:
[0065] Clean outliers, use the majority to replace null values and outliers, use one-hot encoding to encode numeric variables, and use i-bit numeric registers to encode i values of the feature. At this time, the numeric variables will also become sparse matrices to achieve the role of expanding features.
[0066] Machine learning algorithm feature generation module:
[0067] (1) Adaboost new feature generation submodule:
[0068] (a) Divide the data set S into two parts, denoted as S1 and S2.
[0069] (b) Take S1 as the training set, use Adaboost to train the model on the training set, take S2 as the prediction set, use Adaboost to predict S2, and get the probability of S2 samples.
[0070] (c) Set the probability greater than or equal to 0.5 to 1 and less than 0.5 to 0. When AUC is greater than 0.6, save the model.
[0071] (d) The above model is used to predict all samples, and the probability greater than or equal to 0.5 is set to 1, and the probability less than 0.5 is set to 0, and set as a new feature.
[0072] (2) LR new feature generation submodule:
[0073] The same as the adaboost new feature generation submodule, using all samples predicted by the trained LR model, set the probability greater than or equal to 0.5 to 1, and set it to 0 if it is less than 0.5, and set it as a new feature. I will not go into details here.
[0074] Cross-term feature module:
[0075] The FM decomposition machine adds cross features to linear regression, and learns the weights of cross features to obtain the importance of each cross feature, thereby introducing cross-term features.
[0076] (1) Add two feature combinations to linear regression:
[0077]
[0078] Among them, n is the number of features, xi is the value of the i-th feature, and w0, wi, wij are model parameters.
[0079] (2) Introducing matrix decomposition, all quadratic term parameters wij can form a symmetric matrix W, which is then decomposed into the above form, where the jth column of V is the latent vector of the jth dimension feature, that is, each wij =<vi,vj> . Where <> represents the dot product of two vectors.
[0080] (3) After simplifying the quadratic term:
[0081]
[0082] Here k represents the dimension of the latent vector, and f represents the fth element of the vjth latent vector.
[0083] (4) Initialize w and use the logit loss function.
[0084] (5) The parameters w and v in FM are updated by stochastic gradient descent (SGD) to find the optimal solution for w.
[0085] (6) The optimal solution of w is used as the value of the cross-term feature, thereby introducing the second-order feature.
[0086] Weight screening and expansion feature module:
[0087] (1) Perform random forest training on the features obtained from each module to obtain the importance of each feature.
[0088] (2) Filter features with importance less than 0.001.
[0089] (3) The feature with the smallest feature importance is set as 1 variable, and the other features are replicated according to feature importance / min (feature importance) times the number of feature columns.
[0090] The following is a specific example.
[0091] (1) To identify potential users who recharge on special discount days, we used the data from December to build a model, and used the data from January to test the effectiveness of each solution. There are 55 original features of potential users who recharge on special discount days, and the effect of using the original features to train the model is not good.
[0092] (2) Different feature engineering methods:
[0093] The first method is to use manual experience to screen the December data, use statistical methods, segment the continuous data according to the threshold, and then encode it, expand the features, and use correlation and importance detection on the obtained features and the original features. Select one of the two features with greater correlation, remove the feature with less importance, and use the GBDT algorithm to train the obtained features.
[0094] The second method: using the high-dimensional feature engineering algorithm in this embodiment. The specific process is as follows:
[0095] (a) Preprocess the data, perform one-hot encoding on all features, and set all features as dummy variables.
[0096] (b) Use adaboost and LR algorithm to predict samples and generate labels as new features.
[0097] (c) The cross-term features are calculated using the FM algorithm.
[0098] (d) Perform RF importance check on the above features, remove features with low importance, and copy and expand the feature column of other features according to their importance.
[0099] (3) Modeling evaluation:
[0100] The data processed by the above two methods are trained by the GBDT algorithm, and the data for January are predicted. Through comparison and verification, it is found that the effect of this embodiment is better than that of the first method.
[0101] Table 1 Evaluation indicators obtained by two feature engineering schemes
[0102]
[0103] This embodiment has significantly improved the precision and recall rates of identifying users who recharged in T+1 month and users who did not recharge in T+1 month.
[0104] In addition, based on the high-dimensional feature engineering of this embodiment, the deep learning algorithm DNN and the machine learning algorithm GBDT are used for training and comparison analysis as follows:
[0105] Table 2 Evaluation indicators obtained by the two algorithm schemes
[0106]
[0107] Compared with the machine learning algorithm, the deep learning algorithm used in this embodiment can significantly improve the precision and recall rates of identifying users who recharged in T+1 month and users who did not recharge in T+1 month.
[0108] Compared with the prior art, this embodiment has the following technical advantages:
[0109] High accuracy. After processing outliers and missing values for numerical and non-numerical features, this embodiment uses one-hot encoding to encode the features, uses i-bit attribute registers to encode i attributes or i values of the features, generates a high-dimensional sparse matrix, and subsequently uses the FM algorithm to generate cross-term features, which ensures information integrity while fully exploring the potential correlation between features, thereby improving the accuracy of the model.
[0110] The feature discrimination is significant. This embodiment multiplies the importance of features to enhance the significance of important features in high-dimensional features, solving the problem that important features are masked by unimportant features in machine learning algorithms, resulting in poor model effects.
[0111] Applicable to deep learning. The features generated by the feature engineering method in this embodiment are applicable not only to machine learning algorithms, but also to deep learning algorithms.
[0112] Figure 3 : is a schematic diagram of the structure of a high-dimensional feature acquisition device for data provided by an embodiment of the present application, wherein the high-dimensional feature acquisition device for data comprises:
[0113] An acquisition module 301 is used to acquire original data;
[0114] A preprocessing module 302 is used to preprocess the original data to obtain processed data;
[0115] An encoding module 303, used to perform one-hot encoding on the processed data to obtain a first feature;
[0116] A generating module 304, configured to generate a second feature using the processed data based on a machine learning algorithm;
[0117] A merging module 305 is used to merge the first feature and the second feature, and determine whether the importance of the merged feature meets a preset importance threshold;
[0118] The copying and expanding module 306 is used to copy and expand the merged features to obtain high-dimensional features when it is determined that the preset importance threshold is met.
[0119] In one embodiment, the pre-processing module 302 includes:
[0120] The preprocessing unit is used to delete, fill in the mode and clean the original data to obtain processed data.
[0121] In one embodiment, the encoding module 303 includes:
[0122] The encoding unit is used to encode each attribute of the processed data using at least one attribute register to obtain a first feature.
[0123] In one embodiment, the merging module 305 includes:
[0124] A merging unit, used to merge the first feature and the second feature, and determine the importance of the merged feature using a random forest algorithm;
[0125] The judging unit is used to judge whether the importance of each merged feature meets a preset importance threshold.
[0126] Figure 3 Each module / unit in the device shown has the function of realizing Figure 1 The functions of each step in the process can achieve the corresponding technical effects, which will not be described in detail here for the sake of brevity.
[0127] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.
[0128] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.
[0129] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0130] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an electronic device. In a particular embodiment, memory 402 may be a non-volatile solid-state memory.
[0131] In one example, the memory 402 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0132] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any one of the methods for obtaining high-dimensional features of data in the above embodiments.
[0133] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.
[0134] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0135] Bus 410 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.
[0136] In addition, the embodiments of the present application may be implemented by providing a computer storage medium having computer program instructions stored thereon; when the computer program instructions are executed by a processor, any method for obtaining high-dimensional features of data in the above embodiments is implemented.
[0137] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0138] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0139] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or they can be different from the order in the embodiment, or several steps can be performed simultaneously. The above reference is made to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application to describe various aspects of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of a computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, can also be implemented by dedicated hardware that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.
[0140] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A method for obtaining high-dimensional features of data, characterized in that: include: Get the original data; Preprocess the original data to obtain processed data; Performing one-hot encoding on the processed data to obtain a first feature; generating a second feature using the processed data based on a machine learning algorithm; The cross-term features are calculated using the FM algorithm; Merging the first feature, the second feature and the cross-term feature, and determining whether the importance of the merged feature meets a preset importance threshold; When it is determined that the preset importance threshold is met, copying and expanding the merged features to obtain high-dimensional features; The step of performing one-hot encoding on the processed data to obtain the first feature includes: For non-numeric variables, one-hot encoding is used to encode the non-numeric variables, and i-bit attribute registers are used to encode i attributes of the feature. For numeric variables, one-hot encoding is used to encode the numeric variables, and i-bit value registers are used to encode i values of the feature to obtain the first feature; Wherein, based on the machine learning algorithm, using the processed data, generating the second feature includes: Divide the data set S into two parts, record them as S1 and S2, use S1 as the training set, use Adaboost to train the model on the training set, use S2 as the prediction set, use Adaboost to predict S2, get the probability of S2 samples, set the probability greater than or equal to 0.5 to 1, and set it to 0 if it is less than 0.
5. When AUC is greater than 0.6, save the model and use the saved model to predict the full amount of samples, set the probability greater than or equal to 0.5 to 1, and set it to 0 if it is less than 0.5, and set it as a new feature; Use all samples predicted by the trained LR model, set the probability greater than or equal to 0.5 to 1, and set the probability less than 0.5 to 0, and set it as a new feature.
2. The method for obtaining high-dimensional features of data according to claim 1, characterized in that: The preprocessing of the original data to obtain processed data includes: At least one of deleting, filling the mode and cleaning the original data is performed on the outliers and missing values to obtain the processed data.
3. The method for obtaining high-dimensional features of data according to claim 1, characterized in that: The combining of the first feature, the second feature and the cross-term feature, and determining whether the importance of the combined feature meets a preset importance threshold, includes: Merging the first feature, the second feature and the cross-term feature, and determining the importance of the merged feature using a random forest algorithm; It is determined whether the importance of each merged feature meets the preset importance threshold.
4. A device for acquiring high-dimensional features of data, characterized in that: include: An acquisition module is used to obtain raw data; A preprocessing module is used to preprocess the original data to obtain processed data; An encoding module, used for performing one-hot encoding on the processed data to obtain a first feature; A generating module, configured to generate a second feature using the processed data based on a machine learning algorithm; The cross-term features are calculated using the FM algorithm; A merging module, used to merge the first feature, the second feature and the cross-term feature, and determine whether the importance of the merged feature meets a preset importance threshold; A copying and expanding module, used for copying and expanding the merged features to obtain high-dimensional features when it is determined that the preset importance threshold is met; The encoding module is specifically used to encode non-numeric variables using one-hot encoding, encode i attributes of the feature using an i-bit attribute register, encode numeric variables using one-hot encoding, and encode i values of the feature using an i-bit numeric register to obtain the first feature; Wherein, the generation module is specifically used for: Divide the data set S into two parts, record them as S1 and S2, use S1 as the training set, use Adaboost to train the model on the training set, use S2 as the prediction set, use Adaboost to predict S2, get the probability of S2 samples, set the probability greater than or equal to 0.5 to 1, and set it to 0 if it is less than 0.
5. When AUC is greater than 0.6, save the model and use the saved model to predict the full amount of samples, set the probability greater than or equal to 0.5 to 1, and set it to 0 if it is less than 0.5, and set it as a new feature; Use all samples predicted by the trained LR model, set the probability greater than or equal to 0.5 to 1, and set the probability less than 0.5 to 0, and set it as a new feature.
5. The high-dimensional feature acquisition device for data according to claim 4, characterized in that: The preprocessing module comprises: The preprocessing unit is used to delete, fill in the mode and clean the original data to obtain the processed data.
6. The high-dimensional feature acquisition device for data according to claim 4, characterized in that: The merging module comprises: A merging unit, used to merge the first feature, the second feature and the cross-term feature, and determine the importance of the merged features using a random forest algorithm; The judging unit is used to judge whether the importance of each merged feature meets the preset importance threshold.
7. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for obtaining high-dimensional features of data as described in any one of claims 1 to 3 is implemented.
8. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, which, when executed by a processor, implement the method for obtaining high-dimensional features of data as described in any one of claims 1 to 3.
Citation Information
Patent Citations
Feature selection method, terminal, readable medium and computer program
CN109784365A
Video preference prediction method and device, equipment and readable storage medium
CN110059219A
Data processing method and device, computer equipment and storage medium
CN110880014A
Machine learning modeling platform
CN110991474A
Battery replacement prediction method and system and storage medium
CN111191839A
