Method for generating low-frequency features in batches based on high-frequency features

Abstract

The invention relates to the technical field of financial markets, in particular to a method for generating low-frequency features in batches based on high-frequency features, comprising the following steps: S1, acquiring high-frequency feature data from original high-frequency data; s2, selecting part of high-frequency characteristic data in each low-frequency period; s3, aggregating the selected high-frequency features into a value to obtain a feature time sequence of a low-frequency period; and S4, screening out effective low-frequency features from the generated low-frequency features. The invention provides a universal high-frequency feature batch low-frequency process, the step of high-frequency feature low-frequency is refined into two steps of time sampling and aggregation, the low-frequency features are generated in batches in a free combination mode, a time sampling and aggregation expansion method can be designed by oneself, low-frequency feature output is multiplied, and the low-frequency feature generation efficiency is improved. And in combination with feature selection, effective features are directly output, so that the efficiency of the whole process is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of financial markets, and in particular to a method for batch generating low-frequency features from high-frequency features. Background Art

[0002] The market conditions of financial targets are divided into time series data of different frequencies, including different time periods from milliseconds to years. The features extracted from a certain original high-frequency data are often also its original frequency, and there is no good method for converting high-frequency features to low-frequency features. In the prior art, down-converting the second-frequency features to the hourly frequency is to use common aggregation functions such as averaging and summing the second-frequency data per hour to calculate a value as the hourly level feature. There are also some low-frequency features that will filter the high-frequency features of some time periods for aggregation, but generally a fixed method is used to process specific high-frequency features. These methods are inefficient, manual design requires a lot of time, and the number of low-frequency features produced is small. In view of the above problems, the present invention proposes a method for batch generating low-frequency features from high-frequency features to solve the above problems. Summary of the invention

[0003] The purpose of the present invention is to solve the defects mentioned in the above background technology by proposing a method for batch generating low-frequency features from high-frequency features.

[0004] The technical solution adopted by the present invention is as follows:

[0005] A method for batch generating low-frequency features from high-frequency features is provided, comprising the following steps:

[0006] S1: Obtain high-frequency feature data from original high-frequency data;

[0007] S2: Select some high-frequency feature data in each low-frequency period;

[0008] S3: Aggregate the selected high-frequency features into one value to obtain the feature time series of the low-frequency period;

[0009] S4: Filter out effective low-frequency features from the generated low-frequency features.

[0010] As a preferred technical solution of the present invention: the specific steps of S1 are as follows:

[0011] S11: preprocessing time series data;

[0012] S12: determining a high-frequency threshold for distinguishing high-frequency features from low-frequency features;

[0013] S13: extract high-frequency features from the data according to the high-frequency threshold.

[0014] As a preferred technical solution of the present invention: the preprocessing of the time series in S11 includes processing missing data values, processing data duplications and processing data anomalies.

[0015] As a preferred technical solution of the present invention: the specific steps of S2 are as follows:

[0016] S21: define low-frequency features and high-frequency features;

[0017] S22: Select the appropriate sampling rate according to the requirements;

[0018] S23: determining a sampling point in each low-frequency cycle according to the selected sampling rate and specific characteristic data;

[0019] S24: According to the determined sampling point, a corresponding value is selected from the high-frequency feature data as a sampling result;

[0020] S25: Combine the low-frequency features and the sampled high-frequency features into a new data set;

[0021] S26: Analyze the merged dataset to evaluate the validity and quality of the time sampling results.

[0022] As a preferred technical solution of the present invention: the specific steps of S3 are as follows:

[0023] S31: Select the appropriate aggregation method based on the actual data situation and business needs;

[0024] S32: Aggregate the high-frequency features within each low-frequency period using the selected aggregation method;

[0025] S33: merging the aggregated high-frequency features with the original low-frequency features to generate time series data containing the low-frequency features;

[0026] S34: Evaluate the aggregated low-frequency time series.

[0027] As a preferred technical solution of the present invention: the aggregation method in S31 is time-weighted average, and the parameters of the time-weighted average include weighted time interval and average value, and the calculation formula is as follows:

[0028]

[0029] Where Twa is the time-weighted average, t is the time, dt is the length of the time interval, and C(t) is the data value at time t.

[0030] As a preferred technical solution of the present invention: the specific steps of S4 are as follows:

[0031] S41: preprocessing low-frequency features before screening;

[0032] S42: Use the correlation matrix to view the correlation between features;

[0033] S43: performing feature selection on the generated low-frequency features;

[0034] S44: Use another part of the data not used for feature selection for model training and verification;

[0035] S45: Use the cross-validation method for training and verification.

[0036] As a preferred technical solution of the present invention: the correlation between the measurement features in S42 is measured using the Pearson correlation coefficient, and the calculation formula of the Pearson correlation coefficient is as follows:

[0037] It is known that there are n data pairs (a i ,b i ), i = 1, 2, ..., n, a and b are two different features, then the calculation formula of the Pearson correlation coefficient r of feature a and feature b is:

[0038]

[0039] in,

[0040] As a preferred technical solution of the present invention: the feature selection in S43 is based on recursive feature elimination, and the process of the recursive feature elimination is as follows:

[0041] Select the best or worst feature based on the Pearson correlation coefficient;

[0042] Remove selected features from the feature set;

[0043] Repeat the above steps for the remaining feature sets until all features are traversed.

[0044] As a preferred technical solution of the present invention: the criteria for selecting effective low-frequency features from the generated low-frequency features in S4 include:

[0045] The top n tags with the largest Pearson correlations;

[0046] There are no missing values ​​for low-frequency features;

[0047] The standard deviation of low-frequency features is greater than 0.

[0048] The method for batch generating low-frequency features from high-frequency features provided by the present invention has the following beneficial effects compared with the prior art:

[0049] The present invention proposes a universal process for batch low-frequency conversion of high-frequency features, which refines the steps of low-frequency conversion of high-frequency features into two steps: time sampling and aggregation. Low-frequency features are generated in batches through free combination, and the time sampling and aggregation methods can be designed and expanded by themselves to increase the output of low-frequency features exponentially. In addition, the effective features can be directly output in combination with feature selection, which greatly improves the efficiency of the entire process. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this embodiment can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Reference Figure 1 The present invention provides a method for batch generating low-frequency features from high-frequency features, comprising the following steps:

[0053] S1: Obtain high-frequency feature data from original high-frequency data;

[0054] S2: Select some high-frequency feature data in each low-frequency period;

[0055] S3: Aggregate the selected high-frequency features into one value to obtain the feature time series of the low-frequency period;

[0056] S4: Filter out effective low-frequency features from the generated low-frequency features.

[0057] The specific steps of S1 are as follows:

[0058] S11: preprocessing time series data;

[0059] S12: determining a high-frequency threshold for distinguishing high-frequency features from low-frequency features;

[0060] S13: extract high-frequency features from the data according to the high-frequency threshold.

[0061] The preprocessing of the time series in S11 includes processing missing data values, processing data duplications and processing data anomalies.

[0062] The specific steps of S2 are as follows:

[0063] S21: define low-frequency features and high-frequency features;

[0064] S22: Select the appropriate sampling rate according to the requirements;

[0065] S23: determining a sampling point in each low-frequency cycle according to the selected sampling rate and specific characteristic data;

[0066] S24: According to the determined sampling point, a corresponding value is selected from the high-frequency feature data as a sampling result;

[0067] S25: Combine the low-frequency features and the sampled high-frequency features into a new data set;

[0068] S26: Analyze the merged dataset to evaluate the validity and quality of the time sampling results.

[0069] The specific steps of S3 are as follows:

[0070] S31: Select the appropriate aggregation method based on the actual data situation and business needs;

[0071] S32: Aggregate the high-frequency features within each low-frequency period using the selected aggregation method;

[0072] S33: merging the aggregated high-frequency features with the original low-frequency features to generate time series data containing the low-frequency features;

[0073] S34: Evaluate the aggregated low-frequency time series.

[0074] The aggregation method in S31 is time-weighted average, and the parameters of the time-weighted average include weighted time interval and average value. The calculation formula is as follows:

[0075]

[0076] Where Twa is the time-weighted average, t is the time, dt is the length of the time interval, and C(t) is the data value at time t.

[0077] The specific steps of S4 are as follows:

[0078] S41: preprocessing low-frequency features before screening;

[0079] S42: Use the correlation matrix to view the correlation between features;

[0080] S43: performing feature selection on the generated low-frequency features;

[0081] S44: Use another part of the data not used for feature selection for model training and verification;

[0082] S45: Use the cross-validation method for training and verification.

[0083] The correlation between the measurement features in S42 is measured using the Pearson correlation coefficient, and the calculation formula of the Pearson correlation coefficient is as follows:

[0084] It is known that there are n data pairs (a i ,b i ), i = 1, 2, ..., n, a and b are two different features, then the calculation formula of the Pearson correlation coefficient r of feature a and feature b is:

[0085]

[0086] in,

[0087] The feature selection in S43 is based on recursive feature elimination, and the process of recursive feature elimination is as follows:

[0088] Select the best or worst feature based on the Pearson correlation coefficient;

[0089] Remove selected features from the feature set;

[0090] Repeat the above steps for the remaining feature sets until all features are traversed.

[0091] The criteria for selecting effective low-frequency features from the generated low-frequency features in S4 include:

[0092] The top n tags with the largest Pearson correlations;

[0093] There are no missing values ​​for low-frequency features;

[0094] The standard deviation of low-frequency features is greater than 0.

[0095] In this embodiment, high-frequency feature data is obtained from the original high-frequency data by first preprocessing the time series data. The preprocessing includes processing data missing, data duplication, and data anomalies, which can improve the quality of the data, enhance the performance and accuracy of the model, and improve the efficiency of data analysis. High-frequency features usually refer to features with unequal time intervals. Therefore, it is necessary to determine a high-frequency threshold to distinguish high-frequency features from low-frequency features. This threshold may need to be determined based on specific business needs and data conditions. According to the high-frequency threshold, high-frequency features are extracted from the data. These features may include price changes, changes in trading volume, etc. The specific extraction method also needs to be determined based on the data type and business needs.

[0096] Secondly, select some high-frequency feature data in each low-frequency cycle. The specific steps are as follows: first define low-frequency features and high-frequency features. Low-frequency features represent periodic changes or trends, while high-frequency features represent details or transient changes. Select the appropriate sampling rate according to your needs. The sampling rate determines the amount of high-frequency feature data selected in each low-frequency cycle. A higher sampling rate will retain more details, but will increase the computational and storage overhead; a lower sampling rate will sacrifice some details, but can reduce the computational and storage overhead; based on the determined sampling points, select the corresponding value from the high-frequency feature data as the sampling result. In this way, only a part of the high-frequency feature data is retained in each low-frequency cycle; merge the low-frequency features and the sampled high-frequency features into a new data set, which contains part of the high-frequency feature data in each low-frequency cycle; analyze the merged data set to evaluate the effectiveness and quality of the time sampling results.

[0097] Then, the selected high-frequency features are aggregated into one value to obtain the feature time series of the low-frequency period; first, the appropriate aggregation method is selected according to the actual situation of the data and business needs; the high-frequency features in each low-frequency period are aggregated using the selected aggregation method; the aggregated high-frequency features are merged with the original low-frequency features to generate time series data containing low-frequency features; the aggregated low-frequency time series is evaluated. Among them, the aggregation method is time-weighted average, and the parameters of time-weighted average include weighted time interval and average value. The calculation formula is as follows:

[0098]

[0099] Among them, Twa is the time-weighted average, t is time, dt is the length of the time interval, and C(t) is the data value at time t. This formula is obtained by dividing the time interval into countless small time periods, multiplying the data value measured in each time period by the length of the time period, adding all the results, and finally dividing by the total time length to obtain the time-weighted average.

[0100] Finally, effective low-frequency features are screened out from the generated low-frequency features. The specific steps are: pre-process the low-frequency features before screening, including missing value processing, outlier processing, feature scaling or feature encoding; use the correlation matrix to view the correlation between features, and the Pearson correlation coefficient can be used; perform feature selection on the generated low-frequency features; use another part of the data that is not used for feature selection to train and verify the model to confirm the effectiveness of the selected features, and observe the changes in model performance (such as accuracy, recall rate, F1 score, etc.). If the performance is improved, the selected features are effective; in order to more accurately evaluate the performance of the model and the effectiveness of the selected features, use the cross-validation method for training and verification. By dividing the data into k parts, using k-1 parts of the data for training in turn, and the remaining part of the data for verification, a more stable and reliable evaluation result can be obtained.

[0101] Among them, it is known that there are n data pairs (a i ,b i ), i = 1, 2, ..., n, a and b are two different features, then the calculation formula of the Pearson correlation coefficient r of feature a and feature b is:

[0102]

[0103] in, The value range of Pearson correlation coefficient is: |r|≤1, a positive r value indicates that the correlation direction of the two is positively correlated, and a negative r value indicates that the correlation direction is negatively correlated; |r|=1, the two are completely correlated; 0.95≤|r|<1, significant correlation; 0.8≤|r|<0.95, high correlation; 0.5≤|r|<0.8, moderate correlation; 0.3≤|r|<0.5, low correlation; 0<|r|<0.3, weak correlation; |r|=0, no correlation. The sample correlation coefficient is almost the optimal choice for the correlation between variables.

[0104] Furthermore, feature selection is based on recursive feature elimination, which is a greedy algorithm for finding the optimal feature subset. Its main idea is to repeatedly build the model and finally select the best feature set for classification. The process of recursive feature elimination is as follows: select the best or worst feature according to the Pearson correlation coefficient; remove the selected features from the feature set; repeat the above steps in the remaining feature set until all features are traversed. The cyclic process is: train the classifier; calculate the importance measure of the permutation; eliminate irrelevant variables; and train the classifier again with the eliminated features.

[0105] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0106] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for batch generating low-frequency features from high-frequency features, characterized in that: The steps include: S1: Obtain high-frequency feature data from original high-frequency data; S2: Select some high-frequency feature data in each low-frequency period; S3: Aggregate the selected high-frequency features into one value to obtain the feature time series of the low-frequency period; S4: Filter out effective low-frequency features from the generated low-frequency features.

2. The method for batch generating low-frequency features from high-frequency features according to claim 1, characterized in that: The specific steps of S1 are as follows: S11: preprocessing time series data; S12: determining a high-frequency threshold for distinguishing high-frequency features from low-frequency features; S13: extract high-frequency features from the data according to the high-frequency threshold.

3. The method for batch generating low-frequency features from high-frequency features according to claim 2, characterized in that: The preprocessing of the time series in S11 includes processing missing data values, processing data duplications and processing data anomalies.

4. The method for batch generating low-frequency features from high-frequency features according to claim 1, characterized in that: The specific steps of S2 are as follows: S21: define low-frequency features and high-frequency features; S22: Select the appropriate sampling rate according to the requirements; S23: determining a sampling point in each low-frequency cycle according to the selected sampling rate and specific characteristic data; S24: According to the determined sampling point, a corresponding value is selected from the high-frequency feature data as a sampling result; S25: Combine the low-frequency features and the sampled high-frequency features into a new data set; S26: Analyze the merged dataset to evaluate the validity and quality of the time sampling results.

5. The method for batch generating low-frequency features from high-frequency features according to claim 1, characterized in that: The specific steps of S3 are as follows: S31: Select the appropriate aggregation method based on the actual data situation and business needs; S32: Aggregate the high-frequency features within each low-frequency period using the selected aggregation method; S33: merging the aggregated high-frequency features with the original low-frequency features to generate time series data containing the low-frequency features; S34: Evaluate the aggregated low-frequency time series.

6. The method for batch generating low-frequency features from high-frequency features according to claim 5, characterized in that: The aggregation method in S31 is time-weighted average, and the parameters of the time-weighted average include weighted time interval and average value. The calculation formula is as follows: Where Twa is the time-weighted average, t is the time, dt is the length of the time interval, and C(t) is the data value at time t.

7. The method for batch generating low-frequency features from high-frequency features according to claim 1, characterized in that: The specific steps of S4 are as follows: S41: preprocessing low-frequency features before screening; S42: Use the correlation matrix to view the correlation between features; S43: performing feature selection on the generated low-frequency features; S44: Use another part of the data not used for feature selection for model training and verification; S45: Use the cross-validation method for training and verification.

8. The method for batch generating low-frequency features from high-frequency features according to claim 7, characterized in that: The correlation between the measurement features in S42 is measured using the Pearson correlation coefficient, and the calculation formula of the Pearson correlation coefficient is as follows: It is known that there are n data pairs (a i ,b i ), i = 1, 2, ..., n, a and b are two different features, then the calculation formula of the Pearson correlation coefficient r of feature a and feature b is: in, 9. The method for batch generating low-frequency features from high-frequency features according to claim 7, characterized in that: The feature selection in S43 is based on recursive feature elimination, and the process of recursive feature elimination is as follows: Select the best or worst feature based on the Pearson correlation coefficient; Remove selected features from the feature set; Repeat the above steps for the remaining feature sets until all features are traversed.

10. The method for batch generating low-frequency features from high-frequency features according to claim 1, characterized in that: The criteria for selecting effective low-frequency features from the generated low-frequency features in S4 include: The top n tags with the largest Pearson correlations; There are no missing values ​​for low-frequency features; The standard deviation of low-frequency features is greater than 0.

Images