A data analysis system and method based on embedded feature filtering

Abstract

This invention discloses a data analysis system and method based on embedded feature selection, belonging to the field of data analysis technology. The invention includes the following steps: collecting energy consumption data using a combination of high and low frequency methods, continuously recording low-frequency data, detecting suspected peaks based on similarity, and triggering high-frequency sampling; calibrating the timestamps of high and low frequency data, integrating the data, setting peak definition standards, and labeling them; statistically analyzing multiple sets of data features, calculating sampling resilience scores and peak capture indices, and integrating each feature into a complete vector; improving the L1 regularization model, optimizing parameters through cross-validation, selecting features, using LightGBM for prediction, and establishing a dynamic optimization mechanism. This invention can effectively improve the situation where data acquisition is difficult to balance efficiency and accuracy, and where features are redundant during data analysis, in the process of enterprise energy consumption data analysis.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, specifically a data analysis system and method based on embedded feature filtering. Background Technology

[0002] In the field of energy consumption data analysis, accurately capturing peak signals is crucial for monitoring equipment operating status and optimizing energy efficiency. The collection and analysis of energy consumption data directly impacts energy management efficiency, with sampling strategies and feature selection being the core elements determining analytical accuracy and efficiency. While high-frequency sampling can fully preserve data details, it generates a large amount of redundant information, increasing storage and computational costs. Low-frequency sampling, while saving resources, may lead to peak signal loss due to excessively large sampling intervals, affecting the accuracy of anomaly identification. Therefore, balancing data integrity and resource consumption, and designing efficient data analysis methods, has become an urgent problem to be solved.

[0003] Existing energy consumption data analysis techniques mostly employ fixed-frequency sampling strategies, making it difficult to balance efficiency and accuracy. While fixed high-frequency sampling can completely record peak data, it generates a large amount of invalid data during off-peak periods, resulting in significant waste of storage resources. This is especially problematic in long-term monitoring scenarios involving large-scale equipment clusters, where data storage and transmission costs increase significantly. Conversely, fixed low-frequency sampling, while reducing storage pressure, fails to capture instantaneous energy consumption peaks, leading to the omission of key anomaly signals and impacting the accuracy of energy consumption trend analysis and fault diagnosis. Furthermore, traditional methods do not fully consider the impact of sampling frequency on data features during feature selection, potentially introducing unstable features and further reducing the model's prediction accuracy and reliability. Summary of the Invention

[0004] The purpose of this invention is to provide a data analysis system and method based on embedded feature filtering to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data analysis method based on embedded feature filtering, the method comprising the following steps:

[0006] Step 1: Collect energy consumption data using a combination of high and low frequencies. Continuously record low frequencies and detect suspected peak values ​​based on similarity to trigger high-frequency sampling.

[0007] Step 2: Calibrate the high and low frequency data timestamps, integrate the data, set peak value definition standards, and label them;

[0008] Step 3: Statistically analyze the features of multiple sets of data, calculate the sampling resilience score and peak capture index, and integrate the features into a complete vector;

[0009] Step 4: Improve the L1 regularization model, optimize the parameters through cross-validation, select features, use LightGBM for prediction, and establish a dynamic optimization mechanism.

[0010] In step 1, energy consumption data is collected, and low-frequency sampling data of the enterprise's overall equipment group is continuously recorded at a frequency f for daily energy consumption trend monitoring.

[0011] Deploy an edge high-frequency caching module that initiates high-frequency sampling at frequency F only when a suspected peak is detected, with a duration of T. Data is not saved when not triggered to save storage resources, and is recorded as high-frequency sampling data.

[0012] The suspected peak detection method is as follows:

[0013] For low-frequency data collected at intervals of Δf, an estimated sequence X with frequency F is generated by linear interpolation to fill the data gaps in the sampling interval;

[0014] Set the sliding window length τ and calculate the historical pattern similarity at each time t: s(t)=cos(Φ,X[t-τ,t]), where Φ represents the historical typical peak pattern vector, which can be set manually; X[t-τ,t] represents the estimated sequence from (t-τ) to t.

[0015] When the maximum similarity exceeds the threshold θ, high-frequency sampling is started from time t and lasts for a duration of T to capture the true peak signal X(t,t+T).

[0016] In step 2, the timestamps of the low-frequency sampling data and the high-frequency sampling data are calibrated to ensure that the data time base is consistent;

[0017] For periods when high-frequency sampling is not triggered, the interpolated results are used as substitute values ​​and marked as "missing status" to avoid data misuse; low-frequency interpolated data and high-frequency real data are integrated into complete data with a unified high-frequency acquisition frequency;

[0018] Based on high-frequency sampling data X(t,t+T), the peak value is defined as the local maximum value within a set sliding window and the amplitude exceeds the baseline mean by n times the standard deviation; at the same time, whether a peak value occurs is marked.

[0019] In step 3, the low-frequency characteristic group is statistically analyzed: the mean, median, standard deviation, interquartile range, maximum value, and minimum value are calculated for the low-frequency sampling data to reflect the overall trend of energy consumption.

[0020] Statistical analysis of high-frequency feature groups; use real data when high-frequency acquisition is triggered, and use interpolated data when it is not triggered: extract high-frequency mean, standard deviation and maximum value to capture peak details;

[0021] Statistical peak feature group: When marked as having a peak, calculate the peak's rising slope, falling slope, duration, and half-width at half-height to describe the peak's shape; when marked as having no peak, mark the feature group as empty;

[0022] Integrate the various feature groups to form a feature set;

[0023] For any feature m in the feature set j :

[0024] Calculate the sampled toughness score ρ j By simulating different sampling intervals such as 30 seconds, 60 seconds, and 120 seconds, random data point deletion experiments were conducted on historical data to calculate the feature stability: ρ j =1-Std j / (Mean j +ε1); where Std j Mean represents the standard deviation of eigenvalues ​​under different sampling intervals, reflecting the degree of fluctuation; j ε represents the mean of the simulated eigenvalues; ε1 represents the minimum constant to prevent the denominator from being zero; ρ j The closer the value is to 1, the less the feature is affected by fluctuations in the sampling frequency;

[0025] Peak capture index πj: The proportion of statistical characteristics that correctly reflect the true peak value. j =Z j / Sum;

[0026] Among them, Z j Representing feature m j The calculation results accurately reflect the number of events that represent the true peak characteristics, i.e., feature m. j The calculated results are consistent with the actual characteristics of the true peak (such as peak size, rise / fall slope, etc.) or the error is within an acceptable range; Sum represents the total number of true peak events; π j The closer the value is to 1, the stronger the ability to capture feature peaks.

[0027] Each feature m j Integrate into {m jl ,m jh ,ρ j ,π j The complete feature vector of};

[0028] Where, m jl Representing feature m j The feature value calculated based on low-frequency sampling data; m jh Representing feature m j Feature values ​​calculated based on high-frequency sampling data are used to calculate and mark as missing when no trigger is triggered; ρ j Representing feature m jSampling resilience score; π j Representing feature m j Peak capture index.

[0029] In step 4, a double penalty term is added to the L1 regularization model: min w L(w;X,Y)+λΣ j (1+κ1(1-ρ j )+κ2(1-π j ))|w j |;

[0030] Where w represents the feature weight vector; X represents the input feature matrix, with each row corresponding to the low-frequency difference and high-frequency sampling data within the sampling time interval Δf, and each column corresponding to a feature; Y represents the target variable, Y is the binary classification label, used to detect whether a peak occurs; L(w; X, Y) represents the prediction error (e.g., mean squared error); λ represents the regularization strength, controlling feature sparsity; κ1 is the sampling resilience weight, penalizing features sensitive to sampling; κ2 is the peak capture weight, enhancing peak capture capability;

[0031] Multi-fold cross-validation was used to determine the optimal (λ,κ1,κ2) combination, with peak recovery rate and RMSE as evaluation indicators;

[0032] Peak recovery rate = number of correctly captured peaks / total number of true peaks × 100%, which measures the completeness of peak identification by the model;

[0033] Root mean square error RMSE = (1 / M(Σ) i＝1 M (y i -y i ') 2 )) 1 / 2 This reflects the overall prediction bias; where M represents the sample size; y i Represents the true value of the sample; y i 'Indicates the predicted value of the sample;

[0034] Solve the weighted Lasso or GBRT model with L1 regularization using the optimized parameters to generate the feature weight vector w;

[0035] Features whose absolute weights are greater than the threshold ε2 are retained to form the optimal feature subset;

[0036] LightGBM is used as the base model. The optimal feature subset after filtering is input, and the predicted value and probability of peak appear are output. The gradient boosting algorithm is used to optimize the nonlinear mapping capability.

[0037] Set the feature aging period and recalculate ρ based on the new data. j and π jEstablish a dynamic optimization mechanism.

[0038] A data analysis system based on embedded feature filtering, comprising a data acquisition module, a data integration module, a feature engineering module, and a model optimization module;

[0039] The data acquisition module is used to collect energy consumption data in a combination of high and low frequencies. Low-frequency data is continuously recorded, and suspected peak values ​​are detected based on similarity to trigger high-frequency sampling.

[0040] The data integration module is used to calibrate high and low frequency data timestamps, integrate data, set peak definition standards, and label them.

[0041] The feature engineering module is used to statistically analyze multiple sets of data features, calculate the sampling resilience score and peak capture index, and integrate each feature into a complete vector.

[0042] The model optimization module is used to improve the L1 regularized model. It optimizes parameters through cross-validation, selects features, uses LightGBM for prediction, and establishes a dynamic optimization mechanism.

[0043] The data acquisition module includes a low-frequency acquisition unit, a peak detection unit, and a high-frequency triggering unit;

[0044] The low-frequency acquisition unit is used to continuously acquire energy consumption data of the device group at a set frequency to generate a basic time series; the peak detection unit is used to generate a high-frequency estimation sequence through linear interpolation, calculate the similarity of historical patterns, and identify suspected peaks; the high-frequency triggering unit is used to start high-frequency sampling when the similarity is greater than a preset threshold to capture the real peak signal.

[0045] The data integration module includes a timing calibration unit, a data fusion unit, and a peak labeling unit;

[0046] The timing calibration unit is used to align the timestamps of low-frequency and high-frequency data to unify the time reference; the data fusion unit is used to integrate low-frequency interpolated data and high-frequency real data to prevent the missing time period markers from being triggered; the peak labeling unit is used to define peak standards based on a sliding window and perform binary labeling.

[0047] The feature engineering module includes a feature extraction unit, a resilience assessment unit, a vector integration unit, and a capture assessment unit.

[0048] The feature extraction unit is used to calculate low-frequency features, high-frequency features, and peak features; the resilience assessment unit is used to calculate the sampling resilience score through random point deletion experiments; the capture assessment unit is used to calculate the peak capture index; and the vector integration unit is used to generate a complete feature vector.

[0049] The model optimization module includes a regularization optimization unit, a feature selection unit, a prediction modeling unit, and a dynamic update unit;

[0050] The regularization optimization unit is used to improve the L1 regularization model; the feature selection unit is used to solve the weighted Lasso model and retain features with weights not less than a set threshold to form the optimal subset; the prediction modeling unit is used to build a prediction model based on LightGBM and output the peak probability and classification results; the dynamic update unit is used to periodically recalculate the sampling resilience score and peak capture index, and update the feature weights and model parameters.

[0051] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention adopts a low-frequency continuous monitoring combined with high-frequency triggered sampling method, avoiding the waste of storage resources caused by traditional fixed high-frequency sampling, and solving the problem that low-frequency sampling may miss peak signals; This invention integrates low-frequency trend features (mean, standard deviation, etc.), high-frequency detail features (peak slope, duration, etc.) and quantitative indicators (sampling resilience score, peak capture index), which more comprehensively reflects the essence of energy consumption data compared with single-dimensional features; This invention introduces penalty terms for sampling resilience and peak capture index in L1 regularization, giving priority to retaining features that are not sensitive to sampling and have strong peak representation ability, solving the problem that traditional regularization methods may ignore the physical meaning of features, and making the model more focused on key peak features. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating the steps of a data analysis method based on embedded feature filtering according to the present invention;

[0053] Figure 2 This is a flowchart illustrating a data analysis system based on embedded feature filtering according to the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example: Figures 1-2 As shown, the present invention provides a technical solution, a data analysis method based on embedded feature filtering, which includes the following steps:

[0056] Step 1: Collect energy consumption data using a combination of high and low frequencies. Continuously record low frequencies and detect suspected peak values ​​based on similarity to trigger high-frequency sampling.

[0057] Step 2: Calibrate the high and low frequency data timestamps, integrate the data, set peak value definition standards, and label them;

[0058] Step 3: Statistically analyze the features of multiple sets of data, calculate the sampling resilience score and peak capture index, and integrate the features into a complete vector;

[0059] Step 4: Improve the L1 regularization model, optimize the parameters through cross-validation, select features, use LightGBM for prediction, and establish a dynamic optimization mechanism.

[0060] In step 1, energy consumption data is collected, and low-frequency sampling data of the enterprise's overall equipment group is continuously recorded at a frequency f for daily energy consumption trend monitoring.

[0061] Deploy an edge high-frequency caching module that initiates high-frequency sampling at frequency F only when a suspected peak is detected, with a duration of T. Data is not saved when not triggered to save storage resources, and is recorded as high-frequency sampling data.

[0062] The suspected peak detection method is as follows:

[0063] For low-frequency data collected at intervals of Δf, an estimated sequence X with frequency F is generated by linear interpolation to fill the data gaps in the sampling interval;

[0064] Set the sliding window length τ and calculate the historical pattern similarity at each time t: s(t)=cos(Φ,X[t-τ,t]), where Φ represents the historical typical peak pattern vector, which can be set manually; X[t-τ,t] represents the estimated sequence from (t-τ) to t.

[0065] When the maximum similarity exceeds the threshold θ, high-frequency sampling is started from time t and lasts for a duration of T to capture the true peak signal X(t,t+T).

[0066] In step 2, the timestamps of the low-frequency sampling data and the high-frequency sampling data are calibrated to ensure that the data time base is consistent;

[0067] For periods when high-frequency sampling is not triggered, the interpolated results are used as substitute values ​​and marked as "missing status" to avoid data misuse; low-frequency interpolated data and high-frequency real data are integrated into complete data with a unified high-frequency acquisition frequency;

[0068] Based on high-frequency sampling data X(t,t+T), the peak value is defined as the local maximum value within a set sliding window and the amplitude exceeds the baseline mean by n times the standard deviation; at the same time, whether a peak value occurs is marked.

[0069] In step 3, the low-frequency characteristic group is statistically analyzed: the mean, median, standard deviation, interquartile range, maximum value, and minimum value are calculated for the low-frequency sampling data to reflect the overall trend of energy consumption.

[0070] Statistical analysis of high-frequency feature groups; use real data when high-frequency acquisition is triggered, and use interpolated data when it is not triggered: extract high-frequency mean, standard deviation and maximum value to capture peak details;

[0071] Statistical peak feature group: When marked as having a peak, calculate the peak's rising slope, falling slope, duration, and half-width at half-height to describe the peak's shape; when marked as having no peak, mark the feature group as empty;

[0072] Integrate the various feature groups to form a feature set;

[0073] For any feature m in the feature set j :

[0074] Calculate the sampled toughness score ρ j By simulating different sampling intervals such as 30 seconds, 60 seconds, and 120 seconds, random data point deletion experiments were conducted on historical data to calculate the feature stability: ρ j =1-Std j / (Mean j +ε1); where Std j Mean represents the standard deviation of eigenvalues ​​under different sampling intervals, reflecting the degree of fluctuation; j ε represents the mean of the simulated eigenvalues; ε1 represents the minimum constant to prevent the denominator from being zero; ρ j The closer the value is to 1, the less the feature is affected by fluctuations in the sampling frequency;

[0075] Peak capture index πj: The proportion of statistical characteristics that correctly reflect the true peak value. j =Z j / Sum;

[0076] Among them, Z j Representing feature m j The calculation results accurately reflect the number of events that represent the true peak characteristics, i.e., feature m. j The calculated results are consistent with the actual characteristics of the true peak (such as peak size, rise / fall slope, etc.) or the error is within an acceptable range; Sum represents the total number of true peak events; π j The closer the value is to 1, the stronger the ability to capture feature peaks.

[0077] Each feature m j Integrate into {m jl ,m jh ,ρ j ,π j The complete feature vector of};

[0078] Where, m jl Representing feature m j The feature value calculated based on low-frequency sampling data; mjh Representing feature m j Feature values ​​calculated based on high-frequency sampling data are used to calculate and mark as missing when no trigger is triggered; ρ j Representing feature m j Sampling resilience score; π j Representing feature m j Peak capture index.

[0079] In step 4, a double penalty term is added to the L1 regularization model: min w L(w;X,Y)+λΣ j (1+κ1(1-ρ j )+κ2(1-π j ))|w j |;

[0080] Where w represents the feature weight vector; X represents the input feature matrix, with each row corresponding to the low-frequency difference and high-frequency sampling data within the sampling time interval Δf, and each column corresponding to a feature; Y represents the target variable, Y is the binary classification label, used to detect whether a peak occurs; L(w; X, Y) represents the prediction error (e.g., mean squared error); λ represents the regularization strength, controlling feature sparsity; κ1 is the sampling resilience weight, penalizing features sensitive to sampling; κ2 is the peak capture weight, enhancing peak capture capability;

[0081] Multi-fold cross-validation was used to determine the optimal (λ,κ1,κ2) combination, with peak recovery rate and RMSE as evaluation indicators;

[0082] Peak recovery rate = number of correctly captured peaks / total number of true peaks × 100%, which measures the completeness of peak identification by the model;

[0083] Root mean square error RMSE = (1 / M(Σ) i＝1 M (y i -y i ') 2 )) 1 / 2 This reflects the overall prediction bias; where M represents the sample size; y i Represents the true value of the sample; y i 'Indicates the predicted value of the sample;

[0084] Solve the weighted Lasso or GBRT model with L1 regularization using the optimized parameters to generate the feature weight vector w;

[0085] Features whose absolute weights are greater than the threshold ε2 are retained to form the optimal feature subset;

[0086] LightGBM is used as the base model. The optimal feature subset after filtering is input, and the predicted value and probability of peak appear are output. The gradient boosting algorithm is used to optimize the nonlinear mapping capability.

[0087] Set the feature aging period and recalculate ρ based on the new data. j and π j Establish a dynamic optimization mechanism.

[0088] A data analysis system based on embedded feature filtering, comprising a data acquisition module, a data integration module, a feature engineering module, and a model optimization module;

[0089] The data acquisition module is used to collect energy consumption data in a combination of high and low frequencies. Low-frequency data is continuously recorded, and suspected peak values ​​are detected based on similarity to trigger high-frequency sampling.

[0090] The data integration module is used to calibrate high and low frequency data timestamps, integrate data, set peak definition standards, and label them.

[0091] The feature engineering module is used to statistically analyze multiple sets of data features, calculate the sampling resilience score and peak capture index, and integrate each feature into a complete vector.

[0092] The model optimization module is used to improve the L1 regularized model. It optimizes parameters through cross-validation, selects features, uses LightGBM for prediction, and establishes a dynamic optimization mechanism.

[0093] The data acquisition module includes a low-frequency acquisition unit, a peak detection unit, and a high-frequency triggering unit;

[0094] The low-frequency acquisition unit is used to continuously acquire energy consumption data of the device group at a set frequency to generate a basic time series; the peak detection unit is used to generate a high-frequency estimation sequence through linear interpolation, calculate the similarity of historical patterns, and identify suspected peaks; the high-frequency triggering unit is used to start high-frequency sampling when the similarity is greater than a preset threshold to capture the real peak signal.

[0095] The data integration module includes a timing calibration unit, a data fusion unit, and a peak labeling unit;

[0096] The timing calibration unit is used to align the timestamps of low-frequency and high-frequency data to unify the time reference; the data fusion unit is used to integrate low-frequency interpolated data and high-frequency real data to prevent the missing time period markers from being triggered; the peak labeling unit is used to define peak standards based on a sliding window and perform binary labeling.

[0097] The feature engineering module includes a feature extraction unit, a resilience assessment unit, a vector integration unit, and a capture assessment unit.

[0098] The feature extraction unit is used to calculate low-frequency features, high-frequency features, and peak features; the resilience assessment unit is used to calculate the sampling resilience score through random point deletion experiments; the capture assessment unit is used to calculate the peak capture index; and the vector integration unit is used to generate a complete feature vector.

[0099] The model optimization module includes a regularization optimization unit, a feature selection unit, a prediction modeling unit, and a dynamic update unit;

[0100] The regularization optimization unit is used to improve the L1 regularization model; the feature selection unit is used to solve the weighted Lasso model and retain features with weights not less than a set threshold to form the optimal subset; the prediction modeling unit is used to build a prediction model based on LightGBM and output the peak probability and classification results; the dynamic update unit is used to periodically recalculate the sampling resilience score and peak capture index, and update the feature weights and model parameters.

[0101] In this embodiment, an automotive parts manufacturing company has deployed an energy consumption data analysis system based on embedded feature filtering on its stamping production line. This system is used to monitor the energy consumption of the production line in real time, promptly detect abnormal energy consumption peaks, and assist in energy management and equipment operation and maintenance.

[0102] Step 1: Collect energy consumption data using a combination of high and low frequency methods:

[0103] Smart meters and edge computing terminals are deployed in the power distribution cabinets and core equipment (stamping machines, hydraulic systems, etc.) of the stamping production line. The system continuously collects data at a low frequency of once per minute (frequency f), recording parameters such as total power, current, and voltage for daily energy consumption trend analysis. Simultaneously, the edge terminal has a built-in high-frequency cache module (512MB capacity), which only enables high-frequency sampling when peak detection is triggered.

[0104] Data preprocessing: For low-frequency data with a 1-minute interval (Δf), an estimated sequence X with a 1-second frequency (F) is generated using a cubic spline interpolation algorithm to ensure that the interpolated data is smooth and retains key fluctuations.

[0105] Similarity calculation: Set a 60-second sliding window (τ) and calculate the similarity at each time t in real time with the engineer's predefined historical typical peak pattern vector Φ (trained based on historical data of scenarios such as device startup and load change).

[0106] When the maximum similarity exceeds the threshold (θ) of 0.8, high-frequency sampling is performed for 3 minutes (T) at a frequency of 1 second (F) starting from time t. High-frequency data is not stored during periods that are not triggered.

[0107] Step 2: Calibrate and integrate the data and label the peak values:

[0108] After the edge terminal transmits high- and low-frequency data to the central server, it uses the NTP protocol to calibrate the timestamps to ensure that the time reference error between the two types of data is less than 100ms.

[0109] Data integration: For periods where high-frequency sampling was not triggered, fill the data with interpolated data and mark the status as "virtual data"; unify the low-frequency interpolated data and the high-frequency real data into a complete dataset with a frequency of 1 second.

[0110] Peak value definition: Based on high-frequency data, the peak value is defined as the local maximum value within a 15-second sliding window, and the amplitude exceeds 2.5 times the standard deviation of the baseline mean. The system automatically marks the time period and attributes of the peak value.

[0111] Step 3: Collect statistical features and construct feature vectors:

[0112] Low-frequency characteristic group: Calculate the mean, standard deviation, coefficient of variation, and energy consumption peak-valley difference of 1 minute of low-frequency data to reflect the energy consumption stability and overall trend of the production line.

[0113] High-frequency feature group: When high frequency is triggered, 12 detailed features such as peak amplitude, rise time (time taken from baseline to peak), and peak duration are extracted; when not triggered, interpolated data is used for calculation and labeling.

[0114] Peak feature group: If marked as a peak, calculate the upward slope, downward slope, half-width, and other morphological indicators; if there is no peak, mark it as empty.

[0115] Sampling resilience score (ρj): The feature stability is calculated by simulating random point deletion experiments at sampling intervals of 30 seconds, 90 seconds, and 180 seconds.

[0116] Peak capture index (πj): The degree of matching between the statistical characteristic calculation result and the actual peak parameter.

[0117] Each feature mj is integrated into a four-dimensional vector {mjl,mjh,ρj,πj}.

[0118] Step 4: Feature selection, model prediction, and dynamic optimization:

[0119] Improved regularization model: Introduce a double penalty term in L1 regularization, set κ1 = 0.3 (sampling resilience weight) and κ2 = 0.5 (peak capture weight), and optimize the λ (regularization strength) parameter through 10-fold cross-validation.

[0120] Features with absolute weights > 0.1 (ε2) are retained, and the original features are reduced to 32 dimensions.

[0121] A prediction model is built using LightGBM, which takes a filtered subset of features as input and outputs the probability of peak occurrence.

[0122] The characteristic aging cycle is set to one week. Based on the new data, ρj and πj are recalculated, and the model parameters are automatically updated to adapt to changes in equipment operating conditions.

[0123] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A data analysis method based on embedded feature filtering, characterized in that: The method includes the following steps: Step 1: Collect energy consumption data using a combination of high and low frequencies. Continuously record low frequencies and detect suspected peak values ​​based on similarity to trigger high-frequency sampling. Step 2: Calibrate the high and low frequency data timestamps, integrate the data, set peak value definition standards, and label them; Step 3: Statistically analyze the features of multiple sets of data, calculate the sampling resilience score and peak capture index, and integrate the features into a complete vector; Step 4: Improve the L1 regularization model, optimize the parameters through cross-validation, select features, use LightGBM for prediction, and establish a dynamic optimization mechanism. In step 3, the low-frequency characteristic group is statistically analyzed: the mean, median, standard deviation, interquartile range, maximum value, and minimum value are calculated for the low-frequency sampled data; Statistical analysis of high-frequency feature groups; use real data when high-frequency data collection is triggered, and use interpolated data when it is not triggered: extract high-frequency mean, standard deviation and maximum value; Statistical peak feature group: When marked as having a peak, calculate the peak rise slope, fall slope, duration, and half-width at half-height; when marked as having no peak, mark the feature group as empty; Integrate the various feature groups to form a feature set; For any feature m in the feature set j : Calculate the sampled toughness score ρ j By simulating different sampling intervals and conducting random data deletion experiments on historical data, the characteristic stability ρ is calculated. j =1-Std j / (Mean j +ε1); where j represents the index of any single feature in the feature set; Std j Mean represents the standard deviation of eigenvalues ​​under different sampling intervals in the simulation. j ε1 represents the mean of the simulated eigenvalues; ε1 represents a local constant to prevent the denominator from being zero. Calculate the peak capture index π j The proportion of statistical characteristics that correctly reflect the true peak value: π j =Z j / Sum; where Z j Representing feature m j The calculation results accurately reflect the number of events that represent the true peak characteristics; Sum represents the total number of true peak events. Each feature m j Integrate into {m jl ,m jh ,ρ j ,π j The complete feature vector of}; Where, m jl Representing feature m j The feature value calculated based on low-frequency sampling data; m jh Representing feature m j Feature values ​​calculated based on high-frequency sampling data are used to calculate and mark as missing when no trigger is triggered; ρ j Representing feature m j Sampling resilience score; π j Representing feature m j Peak capture index.

2. The data analysis method based on embedded feature filtering according to claim 1, characterized in that: In step 1, energy consumption data is collected, and low-frequency sampling data of the entire enterprise's equipment group is continuously recorded at a frequency f; When a suspected peak is detected, high-frequency sampling with a frequency of F is initiated, and the duration is set to T, which is then recorded as high-frequency sampling data. The suspected peak detection method is as follows: For low-frequency data collected at time intervals of Δf, an estimated sequence X with frequency F is generated by linear interpolation. Set the sliding window length τ, and calculate the historical pattern similarity at each time t: s(t)=cos(Φ,X[t-τ,t]), where Φ represents the historical typical peak pattern vector; X[t-τ,t] represents the estimated sequence from (t-τ) to t. When the maximum similarity exceeds the threshold θ, high-frequency sampling is started from time t and lasts for a duration of T to capture the true peak signal X(t,t+T).

3. The data analysis method based on embedded feature filtering according to claim 2, characterized in that: In step 2, the timestamps of the low-frequency sampling data and the high-frequency sampling data are calibrated to ensure that the data time base is consistent; For periods when high-frequency sampling was not triggered, the interpolated results were used as substitute values ​​and marked as missing; the low-frequency interpolated data and the high-frequency real data were integrated into complete data with a unified high-frequency acquisition frequency. Based on high-frequency sampling data X(t,t+T), the peak value is defined as the local maximum value within a set sliding window and the amplitude exceeds the baseline mean by n times the standard deviation; at the same time, whether a peak value occurs is marked.

4. The data analysis method based on embedded feature filtering according to claim 3, characterized in that: In step 4, a double penalty term is added to the L1 regularization model: min w L(w;X,Y)+λΣ j (1+κ1(1-ρ j )+κ2(1-π j ))|w j |; Where w represents the feature weight vector, w j Representing feature m j The corresponding weights are: X represents the input feature matrix, where each row corresponds to the low-frequency difference and high-frequency sampling data within the sampling time interval Δf, and each column corresponds to a feature; Y is the binary classification target variable indicating whether a peak occurs; L(w;X,Y) represents the prediction error; λ represents the regularization strength; κ1 is the sampling resilience weight; κ2 is the peak capture weight. Multi-fold cross-validation was used to determine the optimal (λ,κ1,κ2) combination, with peak recovery rate and RMSE as evaluation indicators; Peak recovery rate = (Number of correctly captured peaks / Total number of actual peaks) × 100%; Root mean square error RMSE = (1 / M(Σ) i=1 M (y i -y i ') 2 )) 1 / 2 Where M represents the number of samples; y i Represents the true value of the sample; y i 'Indicates the predicted value of the sample; Solve the weighted Lasso or GBRT model with L1 regularization using the optimized parameters to generate the feature weight vector w; Features whose absolute weights are greater than the threshold ε2 are retained to form the optimal feature subset; LightGBM is used as the base model. The optimal feature subset after filtering is input, and the predicted value and probability of peak appear are output. The gradient boosting algorithm is used to optimize the nonlinear mapping capability. Set the feature aging period and recalculate ρ based on the new data. j and π j Establish a dynamic optimization mechanism.

5. A data analysis system based on embedded feature filtering, applied to the data analysis method based on embedded feature filtering as described in any one of claims 1-4, characterized in that: The system includes a data acquisition module, a data integration module, a feature engineering module, and a model optimization module; The data acquisition module is used to collect energy consumption data in a combination of high and low frequencies. Low-frequency data is continuously recorded, and suspected peak values ​​are detected based on similarity to trigger high-frequency sampling. The data integration module is used to calibrate high and low frequency data timestamps, integrate data, set peak definition standards, and label them. The feature engineering module is used to statistically analyze multiple sets of data features, calculate the sampling resilience score and peak capture index, and integrate each feature into a complete vector. The model optimization module is used to improve the L1 regularized model. It optimizes parameters through cross-validation, selects features, uses LightGBM for prediction, and establishes a dynamic optimization mechanism.

6. A data analysis system based on embedded feature filtering according to claim 5, characterized in that: The data acquisition module includes a low-frequency acquisition unit, a peak detection unit, and a high-frequency triggering unit; The low-frequency acquisition unit is used to continuously acquire energy consumption data of the device group at a set frequency to generate a basic time series; the peak detection unit is used to generate a high-frequency estimation sequence through linear interpolation, calculate the similarity of historical patterns, and identify suspected peaks; the high-frequency triggering unit is used to start high-frequency sampling when the similarity is greater than a preset threshold to capture the real peak signal.

7. A data analysis system based on embedded feature filtering according to claim 6, characterized in that: The data integration module includes a timing calibration unit, a data fusion unit, and a peak labeling unit; The timing calibration unit is used to align the timestamps of low-frequency and high-frequency data to unify the time base; the data fusion unit is used to integrate low-frequency interpolated data and high-frequency real data, without triggering the missing time period marker state. The peak labeling unit is used to define peak standards based on a sliding window and perform binary labeling.

8. A data analysis system based on embedded feature filtering according to claim 7, characterized in that: The feature engineering module includes a feature extraction unit, a resilience assessment unit, a vector integration unit, and a capture assessment unit. The feature extraction unit is used to calculate low-frequency features, high-frequency features, and peak features; The resilience assessment unit is used to calculate the sampled resilience score through random point deletion experiments; the capture assessment unit is used to calculate the peak capture index; and the vector integration unit is used to generate a complete feature vector.

9. A data analysis system based on embedded feature filtering according to claim 8, characterized in that: The model optimization module includes a regularization optimization unit, a feature selection unit, a prediction modeling unit, and a dynamic update unit; The regularization optimization unit is used to improve the L1 regularization model; the feature selection unit is used to solve the weighted Lasso model and retain features with weights not less than a set threshold to form the optimal subset; the prediction modeling unit is used to build a prediction model based on LightGBM and output the peak probability and classification results; the dynamic update unit is used to periodically recalculate the sampling resilience score and peak capture index, and update the feature weights and model parameters.

Images