A data feature screening method, device, equipment and medium
By configuring the set of test threshold parameters for out-of-sample window data, and combining information indicators and reference features, data features are filtered and grouped, solving the problem of inaccurate data feature filtering and achieving higher filtering accuracy and differentiation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- E FUND MANAGEMENT CO LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-26
AI Technical Summary
In the data mining process, existing technologies often result in data features that are not semantically clear, have unclear priors, and are highly similar to each other. This leads to inaccurate data feature selection and makes it difficult to determine usability and transferability.
By configuring a set of test threshold parameters for out-of-sample window data, multiple information indicators of candidate data features are determined. These are then combined with reference data features for screening. Representative data features are selected using significance combinations and similarity grouping to ensure the accuracy of the screening.
It improves the accuracy of data feature selection, eliminates candidate features with unclear semantics and vague prior knowledge, and ensures the differentiation and accuracy of the selection results.
Smart Images
Figure CN122087403A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data mining and analysis, and in particular to a method, apparatus, device, and medium for screening data features. Background Technology
[0002] In today's information-driven and intelligent world, the amount of unique data in various industries and fields is constantly increasing, leading to a growing demand for data mining, analysis, and modeling. In the process of data mining, the characteristics of data are crucial because they can best summarize the commonalities and individual characteristics among data objects, thereby expressing the specificity of the data space or dataset.
[0003] In recent years, data feature mining has primarily relied on automated algorithms. This involves generating massive amounts of candidate data features from a data space containing numerous operators and fields, thereby improving mining speed and coverage. However, data features obtained through automated algorithm mining often suffer from semantic ambiguity, unclear priors, and high similarity among features. Current technologies, when faced with these situations, rely solely on human experience or simple rule-based judgments to filter and exclude data features, making it difficult to assess the usability and transferability of individual data features. Consequently, feature filtering is inaccurate. Therefore, improving the accuracy of data feature filtering remains a critical technical challenge that needs to be addressed. Summary of the Invention
[0004] This application provides a method, apparatus, device, and medium for screening data features to address the technical problem that the accuracy of data feature screening in existing data mining processes needs to be improved.
[0005] According to a first aspect of the embodiments of this application, a method for filtering data features is provided, comprising: Based on a preset set of test threshold parameters, multiple information indicators for each candidate data feature are determined sequentially; wherein, the set of test threshold parameters is configured based on out-of-sample window data; the multiple information indicators include feature coverage, feature correlation with the out-of-sample window data and the corresponding in-sample window data, data hierarchical correlation, data hierarchical monotonicity, grouping fit under preset grouping, and feature implementation constraints; Based on the multiple information indicators and the set of test threshold parameters, and combined with the reference data features corresponding to each candidate data feature, the multiple candidate data features are filtered to obtain multiple filtered data features; wherein, the reference data features are constructed based on the corresponding candidate data features; Based on the pairwise similarity between the multiple screening data features, the multiple screening data features are grouped to obtain multiple similar feature groups. Then, based on the multiple information indicators, the screening data features in each similar feature group are selected to obtain the representative data features of each similar feature group.
[0006] This application first determines multiple information indicators for each candidate data feature based on the set of test threshold parameters configured from out-of-sample window data. This clarifies and quantifies statistical indicators to a certain extent, providing a data foundation for subsequent feature screening. Then, based on the multiple information indicators and the set of test threshold parameters, multiple screened data features are obtained by combining each candidate data feature with the corresponding constructed reference data features. By screening using reference data features constructed based on the corresponding candidate data features, the reference data features can be used as priors to measure the semantic features such as feature significance and information increment of the corresponding candidate data features. This helps to eliminate candidate data features with unclear semantics or ambiguous priors, improving the accuracy of data feature screening. Finally, the screened data features are grouped into multiple similar feature groups based on similarity, and representative data features in each similar feature group are selected based on multiple information indicators. This eliminates cases where data features are highly similar, ensuring the differentiation between the screened data features and further improving the accuracy of data feature screening.
[0007] In some embodiments of this application, the set of test threshold parameters is configured based on out-of-sample window data, specifically including: Based on the amount and fluctuation range of out-of-sample window data, the significance threshold and screening threshold are configured to obtain a set of test threshold parameters.
[0008] This application configures the significance threshold and screening threshold by adjusting the amount and fluctuation range of out-of-sample window data, resulting in a set of test threshold parameters. This allows for unified configuration using out-of-sample window data, ensuring consistency in standards for testing data characteristics such as significance and screening. This provides a basis for subsequent determination of information indicators and screening data characteristics.
[0009] In some embodiments of this application, the step of filtering the multiple candidate data features based on the multiple information indicators and the set of test threshold parameters, combined with the reference data features corresponding to each candidate data feature, to obtain multiple filtered data features specifically includes: Based on the multiple information indicators and combined with a preset saliency combination, the multiple candidate data features are filtered to obtain multiple first data features; Based on the set of test threshold parameters, a first similarity is calculated between each first data feature and its corresponding reference data feature. Based on the first similarity, the multiple first data features are filtered to obtain multiple filtered data features.
[0010] This application first selects multiple first data features based on multiple information indicators and combines them with saliency. Then, it calculates the first similarity between each first data feature and the reference data feature based on the set of test threshold parameters, thereby selecting multiple selected data features. By selecting reference data features constructed based on the corresponding candidate data features, the reference data features can be used as priors to measure the semantic features such as feature saliency and information increment of the corresponding candidate data features, thereby eliminating candidate data features with unclear semantics and ambiguous priors, and improving the accuracy of data feature selection.
[0011] In some embodiments of this application, the step of filtering the multiple candidate data features based on the multiple information indicators and a preset saliency combination to obtain multiple first data features specifically includes: If the feature correlation, data hierarchical monotonicity, and grouping fit among the multiple information indicators satisfy the saliency combination, then the corresponding candidate data feature is taken as the first data feature.
[0012] This application uses the corresponding candidate data features as the first data features when the feature correlation, data hierarchical monotonicity, and grouping fit meet the saliency combination. It can measure the feature saliency of data features from multiple aspects through multiple different indicators, thereby avoiding errors and omissions caused by a single indicator or conflicting indicators, and improving the accuracy of data feature selection.
[0013] In some embodiments of this application, the step of calculating a first similarity between each first data feature and its corresponding reference data feature based on the set of test threshold parameters, and filtering the plurality of first data features based on the first similarity to obtain a plurality of filtered data features, specifically includes: Based on the feature components of each first data feature, construct the reference data feature corresponding to each first data feature; Calculate the first similarity between each first data feature and its corresponding reference data feature; Based on the set of test threshold parameters, if the first similarity is less than the preset first similarity threshold, then the corresponding first data feature is used as the filter data feature.
[0014] This application first constructs corresponding reference data features based on the feature components of each first data feature, then calculates the first similarity between each first data feature and the corresponding reference data feature, and based on a set of test threshold parameters, uses the corresponding first data feature as the screening data feature when the first similarity is less than the first similarity threshold. This allows the reference data features constructed based on the corresponding candidate data features to be used as priors. By comparing the similarity between the first data features and the corresponding prior reference data features, and selecting the data features with lower similarity as the screening data features, it can ensure that the selected screening data features have an information increment higher than the corresponding reference data features compared to the first data features, thereby ensuring the feature significance of the corresponding screening data features and improving the accuracy of data feature screening.
[0015] In some embodiments of this application, the step of grouping the plurality of filtered data features according to the pairwise similarity between the plurality of filtered data features to obtain a plurality of similar feature groups specifically includes: Calculate the second similarity between each feature of the filtered data; Based on the set of test threshold parameters, the multiple screening data features are grouped. During grouping, several screening data features with a second similarity greater than a preset second similarity threshold are grouped into the same similar feature group, resulting in multiple similar feature groups.
[0016] This application first calculates the second similarity between each selected data feature, and then groups multiple selected data features into multiple similar feature groups based on the set of test threshold parameters. Grouping by the second similarity can divide multiple selected data features into multiple similar feature groups according to the data tendency they express, which can define the range for further selection of data features, ensure the differentiation between the selected data features, and further improve the accuracy of data feature selection.
[0017] In some embodiments of this application, the step of selecting data features from each similar feature group based on the plurality of information indicators to obtain representative data features for each similar feature group specifically includes: Calculate the information index score for each selected data feature in each similar feature group; wherein the information index score is obtained by standardized aggregation of all information indices of the corresponding selected data feature; Based on the information index score, the selected data features in each similar feature group are chosen. The selected data feature with the highest information index score is taken as the representative data feature of the corresponding similar feature group, thus obtaining the representative data feature of each similar feature group.
[0018] This application first calculates the information index score of each screened data feature in each similar feature group by standardizing and aggregating the corresponding information index. Then, it takes the screened data feature with the highest information index score in each similar feature group as the representative data feature in the group. This can eliminate the situation where the data features are highly similar, ensure the differentiation between the screened data features, and further improve the accuracy of data feature screening.
[0019] According to a second aspect of the embodiments of this application, a data feature filtering device is provided, including a feature index determination module, a data feature filtering module, and a feature grouping selection module; The feature indicator determination module is used to sequentially determine multiple information indicators for each candidate data feature based on a preset set of test threshold parameters; wherein, the set of test threshold parameters is configured based on out-of-sample window data; the multiple information indicators include feature coverage, feature correlation with the out-of-sample window data and the corresponding in-sample window data, data hierarchical correlation, data hierarchical monotonicity, grouping fit under preset grouping, and feature implementation constraints; The data feature filtering module is used to filter the multiple candidate data features based on the multiple information indicators and the set of test threshold parameters, combined with the reference data features corresponding to each candidate data feature, to obtain multiple filtered data features; wherein, the reference data features are constructed based on the corresponding candidate data features; The feature grouping selection module is used to group the multiple filtered data features according to the pairwise similarity between the multiple filtered data features to obtain multiple similar feature groups, and to select the filtered data features in each similar feature group according to the multiple information indicators to obtain the representative data features of each similar feature group.
[0020] In some embodiments of this application, the set of test threshold parameters is configured based on out-of-sample window data, specifically including: Based on the amount and fluctuation range of out-of-sample window data, the significance threshold and screening threshold are configured to obtain a set of test threshold parameters.
[0021] In some embodiments of this application, the data feature filtering module includes a saliency filtering unit and a similarity filtering unit; The saliency filtering unit is used to filter the multiple candidate data features based on the multiple information indicators and in combination with a preset saliency combination to obtain multiple first data features. The similarity filtering unit is used to calculate the first similarity between each first data feature and the corresponding reference data feature according to the set of test threshold parameters, and to filter the plurality of first data features according to the first similarity to obtain a plurality of filtered data features.
[0022] In some embodiments of this application, the saliency screening unit includes a saliency screening subunit; the saliency screening subunit is used to select the corresponding candidate data feature as the first data feature if the feature correlation degree, data hierarchical monotonicity and grouping fit degree among the plurality of information indicators satisfy the saliency combination.
[0023] In some embodiments of this application, the similarity filtering unit includes a reference feature construction subunit, a reference similarity calculation subunit, and a similarity filtering subunit; The reference feature construction subunit is used to construct a reference data feature corresponding to each first data feature based on the feature components of each first data feature. The reference similarity calculation subunit is used to calculate the first similarity between each first data feature and the corresponding reference data feature; The similarity filtering subunit is used to select the corresponding first data feature as the filtering data feature if the first similarity is less than a preset first similarity threshold based on the set of test threshold parameters.
[0024] In some embodiments of this application, the feature grouping selection module includes a feature similarity calculation unit and a similar feature grouping unit; The feature similarity calculation unit is used to calculate the second similarity between each filtered data feature; The similarity feature grouping unit is used to group the multiple screening data features based on the set of test threshold parameters. During grouping, several screening data features with a second similarity greater than a preset second similarity threshold are grouped into the same similarity feature group to obtain multiple similarity feature groups.
[0025] In some embodiments of this application, the feature grouping selection module includes an information index scoring calculation unit and a group-representative feature selection unit; The information index scoring calculation unit is used to calculate the information index score of each selected data feature in each similar feature group; wherein, the information index score is obtained by standardized aggregation of all information indicators of the corresponding selected data feature; The representative feature selection unit within the group is used to select the filtered data features in each similar feature group according to the information index score. When selecting, the filtered data feature with the highest information index score is taken as the representative data feature of the corresponding similar feature group, thus obtaining the representative data feature of each similar feature group.
[0026] This application first determines multiple information indicators for each candidate data feature based on the set of test threshold parameters configured from out-of-sample window data. This clarifies and quantifies statistical indicators to a certain extent, providing a data foundation for subsequent feature screening. Then, based on the multiple information indicators and the set of test threshold parameters, multiple screened data features are obtained by combining each candidate data feature with the corresponding constructed reference data features. By screening using reference data features constructed based on the corresponding candidate data features, the reference data features can be used as priors to measure the semantic features such as feature significance and information increment of the corresponding candidate data features. This helps to eliminate candidate data features with unclear semantics or ambiguous priors, improving the accuracy of data feature screening. Finally, the screened data features are grouped into multiple similar feature groups based on similarity, and representative data features in each similar feature group are selected based on multiple information indicators. This eliminates cases where data features are highly similar, ensuring the differentiation between the screened data features and further improving the accuracy of data feature screening.
[0027] According to a third aspect of the embodiments of this application, a computer device is provided, comprising: a processor; a memory; and a computer program stored in the memory and configured to be executed by the processor; wherein the processor executes the computer program to implement a data feature filtering method as described in this application.
[0028] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute a data feature filtering method according to the present application. Attached Figure Description
[0029] Figure 1 This is a flowchart illustrating a data feature filtering method according to certain embodiments of this application. Figure 2 This is a block diagram of a data feature filtering device shown in some embodiments of this application. Detailed Implementation
[0030] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of this application, and should not be construed as limiting the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments shown in this application without inventive effort are within the protection scope of this application.
[0031] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, unless otherwise explicitly specified, "a plurality of" or "several" means two or more.
[0032] Currently, data features automatically mined by algorithms often suffer from semantic ambiguity, unclear priors, and high similarity among features. Existing technologies, when faced with these situations, rely solely on human experience or simple rule-based judgments to filter and exclude data features, making it difficult to assess the usability and transferability of individual data features. Therefore, improving the accuracy of data feature filtering remains a pressing technical problem that needs to be solved.
[0033] Based on the above technical background, please refer to Figure 1 This application provides a method for filtering data features, including steps S101 to S103, each step of which is as follows: Step S101: Based on the preset set of test threshold parameters, determine multiple information indicators for each candidate data feature in sequence; wherein, the set of test threshold parameters is configured based on out-of-sample window data; the multiple information indicators include feature coverage, feature correlation with the out-of-sample window data and the corresponding in-sample window data, data hierarchical correlation, data hierarchical monotonicity, grouping fit under preset grouping, and feature implementation constraints.
[0034] Generally, the data feature selection method mentioned in this application can be applied to any field that requires data mining analysis of numerical data, such as selecting features that are more effective for prediction in the field of machine learning, and selecting features that are more significant in the field of quantitative statistics. When applying it, it is only necessary to divide the data interval into an in-sample window for feature construction and parameter fitting and an out-of-sample window for verifying the stability and generalization ability of the features.
[0035] Specifically, among the multiple information indicators, feature coverage refers to the number of candidate data features with corresponding feature values in the statistical data, or the ratio of this number to the total amount of statistical data; feature correlation refers to the mean and stability of the correlation or rank correlation time series between candidate data features and the target analysis indicator in the statistical data; data stratification correlation refers to the daily average correlation value and equal weight of each stratum obtained by stratifying the statistical data according to the values of candidate data features, as well as the information ratio indicator used to measure the reliability and stability of the data feature stratification discrimination ability, which is the ratio of the mean of group correlation difference to statistical fluctuation; data stratification monotonicity refers to the statistical test based on the difference values of adjacent stratified data, used to characterize whether the correlation curve increases or decreases in an orderly manner with the value of candidate data features, reflecting the regularity of the correlation between candidate data features and the target indicator; grouping fit refers to the indicator compliance rate and curve consistency under single-direction grouping, reflecting the technical fit under specific scenario constraints; feature implementation constraint refers to the grouped data update frequency, used to constrain data processing costs and technical implementation feasibility.
[0036] In some embodiments of this application, the set of test threshold parameters is configured based on out-of-sample window data, specifically including: Based on the amount and fluctuation range of out-of-sample window data, the significance threshold and screening threshold are configured to obtain a set of test threshold parameters.
[0037] Specifically, when the amount of data in the out-of-sample window is less than the preset data amount threshold, or the fluctuation range is greater than the preset fluctuation threshold, the significance threshold in the configured test threshold parameter set will be reduced to reduce the risk of missing data features and improve the accuracy and stability of data feature screening.
[0038] This application configures the significance threshold and screening threshold by adjusting the amount and fluctuation range of out-of-sample window data, resulting in a set of test threshold parameters. This allows for unified configuration using out-of-sample window data, ensuring consistency in standards for testing data characteristics such as significance and screening. This provides a basis for subsequent determination of information indicators and screening data characteristics.
[0039] Step S102: Based on the multiple information indicators and the set of test threshold parameters, and combined with the reference data features corresponding to each candidate data feature, the multiple candidate data features are filtered to obtain multiple filtered data features; wherein, the reference data features are constructed based on the corresponding candidate data features.
[0040] In some embodiments of this application, the step of filtering the multiple candidate data features based on the multiple information indicators and the set of test threshold parameters, combined with the reference data features corresponding to each candidate data feature, to obtain multiple filtered data features specifically includes: Based on the multiple information indicators and combined with a preset saliency combination, the multiple candidate data features are filtered to obtain multiple first data features; Based on the set of test threshold parameters, a first similarity is calculated between each first data feature and its corresponding reference data feature. Based on the first similarity, the multiple first data features are filtered to obtain multiple filtered data features.
[0041] This application first selects multiple first data features based on multiple information indicators and combines them with saliency. Then, it calculates the first similarity between each first data feature and the reference data feature based on the set of test threshold parameters, thereby selecting multiple selected data features. By selecting reference data features constructed based on the corresponding candidate data features, the reference data features can be used as priors to measure the semantic features such as feature saliency and information increment of the corresponding candidate data features, thereby eliminating candidate data features with unclear semantics and ambiguous priors, and improving the accuracy of data feature selection.
[0042] In some embodiments of this application, the step of filtering the multiple candidate data features based on the multiple information indicators and a preset saliency combination to obtain multiple first data features specifically includes: If the feature correlation, data hierarchical monotonicity, and grouping fit among the multiple information indicators satisfy the saliency combination, then the corresponding candidate data feature is taken as the first data feature.
[0043] Specifically, when the signs of the feature correlation between the out-of-sample window data and the in-sample window data are consistent, the data stratification monotonicity satisfies the weakest adjacent difference lower limit, and the grouping fit reaches the scenario fit lower limit, it is considered to satisfy the preset significance combination.
[0044] This application uses the corresponding candidate data features as the first data features when the feature correlation, data hierarchical monotonicity, and grouping fit meet the saliency combination. It can measure the feature saliency of data features from multiple aspects through multiple different indicators, thereby avoiding errors and omissions caused by a single indicator or conflicting indicators, and improving the accuracy of data feature selection.
[0045] In some embodiments of this application, the step of calculating a first similarity between each first data feature and its corresponding reference data feature based on the set of test threshold parameters, and filtering the plurality of first data features based on the first similarity to obtain a plurality of filtered data features, specifically includes: Based on the feature components of each first data feature, construct the reference data feature corresponding to each first data feature; Calculate the first similarity between each first data feature and its corresponding reference data feature; Based on the set of test threshold parameters, if the first similarity is less than the preset first similarity threshold, then the corresponding first data feature is used as the filter data feature.
[0046] Specifically, the feature components of the first data feature can be several basic fields contained in the first data feature, and the reference data feature constructed in this case is a simple sum of these basic fields.
[0047] Specifically, the methods for measuring the first similarity include, but are not limited to, Pearson correlation coefficient, Spearman rank correlation coefficient, or cosine similarity; the first similarity threshold can be configured according to the test threshold parameter set. Generally, since the reference data feature represents the benchmark reference corresponding to the first data feature, when the first similarity between the first data feature and the corresponding reference data feature is not less than the first similarity threshold, the first data feature is considered to lack information increment and have low feature significance. Therefore, first data features with a first similarity not less than the first similarity threshold should be eliminated, while those with a first similarity less than the first similarity threshold should be used as the basis for further screening.
[0048] This application first constructs corresponding reference data features based on the feature components of each first data feature, then calculates the first similarity between each first data feature and the corresponding reference data feature, and based on a set of test threshold parameters, uses the corresponding first data feature as the screening data feature when the first similarity is less than the first similarity threshold. This allows the reference data features constructed based on the corresponding candidate data features to be used as priors. By comparing the similarity between the first data features and the corresponding prior reference data features, and selecting the data features with lower similarity as the screening data features, it can ensure that the selected screening data features have an information increment higher than the corresponding reference data features compared to the first data features, thereby ensuring the feature significance of the corresponding screening data features and improving the accuracy of data feature screening.
[0049] Step S103: Based on the pairwise similarity between the multiple screening data features, group the multiple screening data features to obtain multiple similar feature groups, and select the screening data features in each similar feature group according to the multiple information indicators to obtain the representative data features of each similar feature group.
[0050] In some embodiments of this application, the step of grouping the plurality of filtered data features according to the pairwise similarity between the plurality of filtered data features to obtain a plurality of similar feature groups specifically includes: Calculate the second similarity between each feature of the filtered data; Based on the set of test threshold parameters, the multiple screening data features are grouped. During grouping, several screening data features with a second similarity greater than a preset second similarity threshold are grouped into the same similar feature group, resulting in multiple similar feature groups.
[0051] Specifically, the methods for measuring the second similarity include, but are not limited to, Pearson correlation coefficient, Spearman rank correlation coefficient, or cosine similarity; the second similarity threshold can be configured according to the test threshold parameter set. Generally, since the selected data features within the same similar feature group have a certain degree of similarity, it indicates that all data features within that similar feature group have the same or highly correlated data tendency or data description tendency. Therefore, by dividing similar feature groups, the data features can be further filtered to ensure the differentiation between data features.
[0052] This application first calculates the second similarity between each selected data feature, and then groups multiple selected data features into multiple similar feature groups based on the set of test threshold parameters. Grouping by the second similarity can divide multiple selected data features into multiple similar feature groups according to the data tendency they express, which can define the range for further selection of data features, ensure the differentiation between the selected data features, and further improve the accuracy of data feature selection.
[0053] In some embodiments of this application, the step of selecting data features from each similar feature group based on the plurality of information indicators to obtain representative data features for each similar feature group specifically includes: Calculate the information index score for each selected data feature in each similar feature group; wherein the information index score is obtained by standardized aggregation of all information indices of the corresponding selected data feature; Based on the information index score, the selected data features in each similar feature group are chosen. The selected data feature with the highest information index score is taken as the representative data feature of the corresponding similar feature group, thus obtaining the representative data feature of each similar feature group.
[0054] Specifically, when calculating the information index score of the selected data feature, the information index of all the selected data feature can be standardized and aggregated by means of maximization, weighting or segmentation penalty to obtain the corresponding information index score.
[0055] This application first calculates the information index score of each screened data feature in each similar feature group by standardizing and aggregating the corresponding information index. Then, it takes the screened data feature with the highest information index score in each similar feature group as the representative data feature in the group. This can eliminate the situation where the data features are highly similar, ensure the differentiation between the screened data features, and further improve the accuracy of data feature screening.
[0056] Compared to existing technologies, this application first determines multiple information indicators for each candidate data feature based on the set of test threshold parameters configured from out-of-sample window data. This clarifies and quantifies statistical indicators to a certain extent, providing a data foundation for subsequent feature screening. Then, based on the multiple information indicators and the set of test threshold parameters, multiple screened data features are obtained by combining each candidate data feature with corresponding constructed reference data features. By screening based on the reference data features constructed from the corresponding candidate data features, the reference data features can be used as priors to measure semantic features such as feature significance and information increment of the corresponding candidate data features. This eliminates candidate data features with unclear semantics or ambiguous priors, improving the accuracy of data feature screening. Finally, the screened data features are grouped into multiple similar feature groups based on similarity, and representative data features in each similar feature group are selected based on multiple information indicators. This eliminates cases where data features are highly similar, ensuring the differentiation between the screened data features and further improving the accuracy of data feature screening.
[0057] For a method corresponding to the one described above, please refer to [link to relevant documentation]. Figure 2 This application provides a data feature filtering device, including a feature index determination module 210, a data feature filtering module 220, and a feature grouping selection module 230. The feature index determination module 210 is used to determine multiple information indicators for each candidate data feature in sequence according to a preset set of test threshold parameters; wherein, the set of test threshold parameters is configured based on out-of-sample window data; the multiple information indicators include feature coverage, feature correlation with the out-of-sample window data and the corresponding in-sample window data, data hierarchical correlation, data hierarchical monotonicity, grouping fit under preset grouping, and feature implementation constraints; The data feature filtering module 220 is used to filter the multiple candidate data features based on the multiple information indicators and the set of test threshold parameters, combined with the reference data features corresponding to each candidate data feature, to obtain multiple filtered data features; wherein, the reference data features are constructed based on the corresponding candidate data features; The feature grouping selection module 230 is used to group the multiple filtered data features according to the pairwise similarity between the multiple filtered data features to obtain multiple similar feature groups, and to select the filtered data features in each similar feature group according to the multiple information indicators to obtain the representative data features of each similar feature group.
[0058] In some embodiments of this application, the set of test threshold parameters is configured based on out-of-sample window data, specifically including: Based on the amount and fluctuation range of out-of-sample window data, the significance threshold and screening threshold are configured to obtain a set of test threshold parameters.
[0059] In some embodiments of this application, the data feature filtering module 220 includes a saliency filtering unit and a similarity filtering unit; The saliency filtering unit is used to filter the multiple candidate data features based on the multiple information indicators and in combination with a preset saliency combination to obtain multiple first data features. The similarity filtering unit is used to calculate the first similarity between each first data feature and the corresponding reference data feature according to the set of test threshold parameters, and to filter the plurality of first data features according to the first similarity to obtain a plurality of filtered data features.
[0060] In some embodiments of this application, the saliency screening unit includes a saliency screening subunit; the saliency screening subunit is used to select the corresponding candidate data feature as the first data feature if the feature correlation degree, data hierarchical monotonicity and grouping fit degree among the plurality of information indicators satisfy the saliency combination.
[0061] In some embodiments of this application, the similarity filtering unit includes a reference feature construction subunit, a reference similarity calculation subunit, and a similarity filtering subunit; The reference feature construction subunit is used to construct a reference data feature corresponding to each first data feature based on the feature components of each first data feature. The reference similarity calculation subunit is used to calculate the first similarity between each first data feature and the corresponding reference data feature; The similarity filtering subunit is used to select the corresponding first data feature as the filtering data feature if the first similarity is less than a preset first similarity threshold based on the set of test threshold parameters.
[0062] In some embodiments of this application, the feature grouping selection module 230 includes a feature similarity calculation unit and a similar feature grouping unit; The feature similarity calculation unit is used to calculate the second similarity between each filtered data feature; The similarity feature grouping unit is used to group the multiple screening data features based on the set of test threshold parameters. During grouping, several screening data features with a second similarity greater than a preset second similarity threshold are grouped into the same similarity feature group to obtain multiple similarity feature groups.
[0063] In some embodiments of this application, the feature grouping selection module 230 includes an information index scoring calculation unit and a group representative feature selection unit; The information index scoring calculation unit is used to calculate the information index score of each selected data feature in each similar feature group; wherein, the information index score is obtained by standardized aggregation of all information indicators of the corresponding selected data feature; The representative feature selection unit within the group is used to select the filtered data features in each similar feature group according to the information index score. When selecting, the filtered data feature with the highest information index score is taken as the representative data feature of the corresponding similar feature group, thus obtaining the representative data feature of each similar feature group.
[0064] This application first determines multiple information indicators for each candidate data feature based on the set of test threshold parameters configured from out-of-sample window data. This clarifies and quantifies statistical indicators to a certain extent, providing a data foundation for subsequent feature screening. Then, based on the multiple information indicators and the set of test threshold parameters, multiple screened data features are obtained by combining each candidate data feature with the corresponding constructed reference data features. By screening using reference data features constructed based on the corresponding candidate data features, the reference data features can be used as priors to measure the semantic features such as feature significance and information increment of the corresponding candidate data features. This helps to eliminate candidate data features with unclear semantics or ambiguous priors, improving the accuracy of data feature screening. Finally, the screened data features are grouped into multiple similar feature groups based on similarity, and representative data features in each similar feature group are selected based on multiple information indicators. This eliminates cases where data features are highly similar, ensuring the differentiation between the screened data features and further improving the accuracy of data feature screening.
[0065] It should be understood that the apparatus provided in the embodiments of this application is corresponding to the aforementioned method. The data feature filtering apparatus provided in the embodiments of this application can implement the data feature filtering method provided in any embodiment of this application.
[0066] Adaptively, embodiments of this application also provide a computer device and a computer-readable storage medium.
[0067] The computer device includes: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; The processor executes the computer program to implement a data feature filtering method according to this application.
[0068] The computer-readable storage medium stores multiple instructions adapted for loading by a processor to execute a data feature filtering method according to this application.
[0069] The above description represents some embodiments of this application, providing a further detailed explanation of the purpose, technical solution, and beneficial effects of this application. It should be understood that the above-described embodiments of this application should not be construed as limiting this application. In particular, any changes, modifications, equivalent substitutions, and variations made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for filtering data features, characterized in that, include: Based on a preset set of test threshold parameters, multiple information indicators for each candidate data feature are determined sequentially; wherein, the set of test threshold parameters is configured based on out-of-sample window data; the multiple information indicators include feature coverage, feature correlation with the out-of-sample window data and the corresponding in-sample window data, data hierarchical correlation, data hierarchical monotonicity, grouping fit under preset grouping, and feature implementation constraints; Based on the multiple information indicators and the set of test threshold parameters, and combined with the reference data features corresponding to each candidate data feature, the multiple candidate data features are filtered to obtain multiple filtered data features; wherein, the reference data features are constructed based on the corresponding candidate data features; Based on the pairwise similarity between the multiple screening data features, the multiple screening data features are grouped to obtain multiple similar feature groups. Then, based on the multiple information indicators, the screening data features in each similar feature group are selected to obtain the representative data features of each similar feature group.
2. The data feature filtering method according to claim 1, characterized in that, The set of test threshold parameters is configured based on out-of-sample window data and specifically includes: Based on the amount and fluctuation range of out-of-sample window data, the significance threshold and screening threshold are configured to obtain a set of test threshold parameters.
3. The data feature filtering method according to claim 1, characterized in that, The step of filtering the multiple candidate data features based on the multiple information indicators and the set of test threshold parameters, combined with the reference data features corresponding to each candidate data feature, to obtain multiple filtered data features, specifically includes: Based on the multiple information indicators and combined with a preset saliency combination, the multiple candidate data features are filtered to obtain multiple first data features; Based on the set of test threshold parameters, a first similarity is calculated between each first data feature and its corresponding reference data feature. Based on the first similarity, the multiple first data features are filtered to obtain multiple filtered data features.
4. The data feature filtering method according to claim 3, characterized in that, The step of filtering the multiple candidate data features based on the multiple information indicators and combined with a preset saliency combination to obtain multiple first data features specifically includes: If the feature correlation, data hierarchical monotonicity, and grouping fit among the multiple information indicators satisfy the saliency combination, then the corresponding candidate data feature is taken as the first data feature.
5. The data feature filtering method according to claim 3, characterized in that, The step of calculating a first similarity between each first data feature and its corresponding reference data feature based on the set of test threshold parameters, and then filtering the plurality of first data features based on the first similarity to obtain a plurality of filtered data features, specifically includes: Based on the feature components of each first data feature, construct the reference data feature corresponding to each first data feature; Calculate the first similarity between each first data feature and its corresponding reference data feature; Based on the set of test threshold parameters, if the first similarity is less than the preset first similarity threshold, then the corresponding first data feature is used as the filter data feature.
6. The data feature filtering method according to claim 1, characterized in that, The step of grouping the multiple selected data features based on the pairwise similarity between each other to obtain multiple similar feature groups specifically includes: Calculate the second similarity between each feature of the filtered data; Based on the set of test threshold parameters, the multiple screening data features are grouped. During grouping, several screening data features with a second similarity greater than a preset second similarity threshold are grouped into the same similar feature group, resulting in multiple similar feature groups.
7. The data feature filtering method according to claim 1, characterized in that, The step of selecting data features from each similar feature group based on the multiple information indicators to obtain representative data features for each similar feature group specifically includes: Calculate the information index score for each selected data feature in each similar feature group; wherein the information index score is obtained by standardized aggregation of all information indices of the corresponding selected data feature; Based on the information index score, the selected data features in each similar feature group are chosen. The selected data feature with the highest information index score is taken as the representative data feature of the corresponding similar feature group, thus obtaining the representative data feature of each similar feature group.
8. A data feature filtering device, characterized in that, It includes a feature indicator determination module, a data feature filtering module, and a feature grouping selection module; The feature indicator determination module is used to determine multiple information indicators for each candidate data feature sequentially according to a preset set of test threshold parameters; wherein, the set of test threshold parameters is configured based on out-of-sample window data; the multiple information indicators include feature coverage, feature correlation with the out-of-sample window data and the corresponding in-sample window data, data hierarchical correlation, data hierarchical monotonicity, grouping fit under preset grouping, and feature implementation constraints; The data feature filtering module is used to filter the multiple candidate data features based on the multiple information indicators and the set of test threshold parameters, combined with the reference data features corresponding to each candidate data feature, to obtain multiple filtered data features; wherein, the reference data features are constructed based on the corresponding candidate data features; The feature grouping selection module is used to group the multiple filtered data features according to the pairwise similarity between the multiple filtered data features to obtain multiple similar feature groups, and to select the filtered data features in each similar feature group according to the multiple information indicators to obtain the representative data features of each similar feature group.
9. A computer device, characterized in that, include: processor; Memory; A computer program stored in the memory and configured to be executed by the processor; The processor executes the computer program to implement a data feature filtering method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute a data feature filtering method according to any one of claims 1 to 7.