Energy operation command management system and method based on big data

Through the energy operation command and management system based on big data, using feature selection and evaluation functions, combined with the L1 regularization term, the problem of inefficiency in traditional systems when processing large-scale energy data is solved, and efficient analysis and integration of energy data is achieved. Provides scientific and reliable decision-making support.

CN118940973APending Publication Date: 2024-11-12SHANDONG GUOYAN AUTOMATION CO LTD

Patent Information

Application Number
CN202411419748.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Traditional energy management systems are difficult to effectively process and analyze large-scale, multi-dimensional, and hybrid energy data, resulting in low data integration and analysis efficiency, which cannot meet the needs of modern enterprises for refined management and energy conservation and emission reduction.

Method used

Adopting a big data-based energy operation command and management system, through the use of feature selection and evaluation functions, the important feature subset is automatically selected and verified, and data integration is carried out, combining the loss function of the L1 regularization term for feature selection, and redundant features are eliminated. , ensure the effectiveness and efficiency of data.

Benefits of technology

It improves the analysis and processing capabilities of energy data, ensures the scientificity and representativeness of feature selection, improves algorithm efficiency, avoids resource waste, provides objective and reliable verification results, and provides a strong basis for energy operation decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940973A_ABST
    Figure CN118940973A_ABST
Patent Text Reader

Abstract

The invention discloses an energy operation command management system and method based on big data, and belongs to the technical field of energy operation management, and the method comprises the steps: obtaining energy data, generating a feature selection instruction according to the energy data, executing the instruction, evaluating a feature subset through an evaluation function, and obtaining an evaluation function value; querying a stop criterion related to the evaluation function from a preset database; if the evaluation function value reaches a stop criterion, stopping searching; verifying the actual validity of the feature subset according to the verification data set; if the actual effectiveness reaches a preset effective value, outputting a final feature subset; and according to the final feature subset, generating and executing a data integration instruction. The method has the effect of improving the energy data analysis and processing capability of the energy operation command management system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of energy operation management, and in particular to a big data-based energy operation command and management system and method thereof. Background Art

[0002] Energy management refers to the active and systematic monitoring, control and optimization of an organization's energy consumption, so as to achieve the goal of saving energy consumption and reducing energy costs. With the development of information technology and the growing demand for energy, traditional energy management methods can no longer meet the requirements of modern enterprises for refined management and energy conservation and emission reduction.

[0003] In the era of big data, the types and scale of energy data are rapidly expanding, covering multiple dimensions from energy consumption measurement, environmental monitoring to production operations. However, the dynamic update and change of sample size and dimension greatly increase the computational burden. In these dynamic data, most data samples do not exist in the form of a single data value, but are mixed data containing both symbolic data and numerical data. Energy data has different formats and uneven quality, which leads to huge challenges in data integration and analysis for energy operation command and management systems. Summary of the invention

[0004] In order to improve the energy operation command and management system's ability to analyze and process energy data, the present application provides an energy operation command and management system based on big data and a method thereof.

[0005] The energy operation command and management system based on big data and the method thereof provided in this application adopt the following technical solutions: In a first aspect, the present application provides an energy operation command and management method based on big data, comprising the following steps: Acquiring energy data, wherein the energy data includes a plurality of data units; Generate and execute a feature selection instruction according to the energy data, wherein the feature selection instruction is used to select a feature subset from a plurality of data units, wherein the feature subset includes a plurality of valid features; Using an evaluation function to evaluate the feature subset, to obtain an evaluation function value; Querying a stop criterion related to the evaluation function from a preset database; If the evaluation function value reaches the stopping criterion, the search stops; Querying a verification data set and a preset valid value corresponding to the feature subset from a preset database; Verifying the actual validity of the feature subset according to the verification data set; If the actual validity reaches the preset validity value, the final feature subset is output; According to the final feature subset, a data integration instruction is generated and executed, wherein the data integration instruction is used to integrate the feature subset with other energy data.

[0006] Furthermore, the step of generating and executing the feature selection instruction specifically includes: Define the loss function: ; Among them: p represents the number of features, that is, the total number of independent variables; y i represents the target variable value of the i-th sample, that is, the dependent variable observation value; x ij represents the jth eigenvalue of the i-th sample, that is, the observed value of the independent variable; β j represents the p-dimensional coefficient vector, that is, the model parameters that need to be solved; n represents the number of samples, that is, the total number of observations; λ represents the regularization parameter, which is used to control the strength of regularization to prevent overfitting; Find the optimal coefficient vector β by minimizing the objective function; Due to the L1 regularization term The existence of will gradually reduce the coefficients of unimportant data features to 0 during the iteration process; In the final coefficient vector β, the data features with non-zero coefficients are considered important and used to form the feature subset; while the data features with coefficients compressed to 0 are considered redundant or unimportant and are eliminated from the feature subset.

[0007] Furthermore, the step of finding the optimal coefficient vector β by minimizing the objective function specifically includes: Select the update dimension. In each iteration, select a dimension j (j=1, 2, 3...p) to update. Fix the coefficients of other dimensions β k (k≠j), calculate the residual r of the current dimension j; ; Where: r ij represents the residual of the i-th observation in the j-th dimension, that is, the prediction error after removing the influence of the current dimension; x ik represents the kth feature of the i-th observation; Calculate the part of the objective function in this dimension and replace the β in the objective function j Items are extracted separately and we get: ; Where: L j (β j ) represents the part of the objective function on dimension j, that is, only β is considered j The loss of time; x ijrepresents the jth feature of the i-th observation; Find the optimal solution for L j (β j ), calculate the β that minimizes it j ; When β j >0, for L j (β j ) and set it to 0, and we get: ; like , then the solution is valid; otherwise β j =0; When β j <0, the solution is: ; like , then the solution is valid; otherwise β j =0; When β j =0, if , then the solution is valid; After the iteration is completed, the final coefficient vector β is output, which is the optimal coefficient vector β that minimizes the objective function of LASSO.

[0008] Furthermore, before the step of generating and executing the feature selection instruction, the method further includes: Generate and execute data review instructions based on the energy data, wherein the data review instructions are used to check the energy data and retrieve erroneous data, missing data, and abnormal data; If erroneous data, missing data and / or abnormal data are retrieved, data processing instructions are generated and executed, and the data processing instructions are used to process the erroneous data, missing data, and abnormal data, including processing abnormal values, filling missing values, deleting duplicate records, and correcting errors; Generate and execute data conversion instructions, wherein the data conversion instructions are used to convert data types, process date formats, and normalize or standardize data so that energy data of different magnitudes can be compared and analyzed; Generate and execute a data screening instruction, wherein the data screening instruction is used to screen the processed energy data.

[0009] Furthermore, the step of generating and executing the data screening instruction specifically includes: The data unit is defined as feature f, and the relevance score of feature f is defined as: ; Where: f i represents the value of sample i on feature f; Sij Represents the corresponding value in the weight matrix S. S is a simulated expression of the data space structure, describing the distance between two samples of the same type (neighborhood). The greater the distance between two samples of the same type (neighborhood), the greater the corresponding weight, and vice versa. Var(f) represents the variance of feature f. Query a preset score threshold from a preset database; Among all data units, select the preferred data units whose relevance scores reach the preset score threshold.

[0010] Furthermore, the step of generating and executing the data review instruction specifically includes: Query data processing tools, preset audit rules and audit statistics rules from the preset database; Verify the energy data according to the preset audit rules and audit statistical rules, and identify abnormal data items that do not comply with the preset audit rules and / or audit statistical rules; Marking and storing the abnormal data item and detailed information corresponding to the abnormal data item, wherein the detailed information includes field description, data type, location, error cause, data range, and timestamp; According to the data processing tool, using statistical functions in the data processing tool to detect missing data values ​​in the energy data; By analyzing the distribution and pattern of energy data, missing values ​​that do not conform to the regular pattern are identified; The missing data value and detailed information corresponding to the missing data value are marked and stored.

[0011] Furthermore, after the step of marking and storing the missing data value and the detailed information corresponding to the missing data value, the method further includes: Obtain key indicator data from the preset database; If there is key indicator data in the data missing value, an early warning instruction is generated and executed, and the early warning instruction is used to push the data missing value and detailed information corresponding to the data missing value to the user's smart terminal.

[0012] In the second aspect, the present application provides an energy operation command and management system based on big data, including: An energy data acquisition module, used to acquire energy data, wherein the energy data includes a plurality of data units; A feature selection module, used for generating and executing a feature selection instruction according to the energy data, wherein the feature selection instruction is used for selecting a feature subset from a plurality of data units, wherein the feature subset includes a plurality of valid features; A feature subset evaluation module, used to evaluate the feature subset using an evaluation function to obtain an evaluation function value; A stop criterion query module, used to query a stop criterion related to the evaluation function from a preset database; A search module, used to stop searching if the evaluation function value reaches a stopping criterion; A verification data set query module, used to query a verification data set and a preset valid value corresponding to the feature subset from a preset database; A feature verification module, used to verify the actual validity of the feature subset based on the verification data set; A feature output module, used for outputting a final feature subset if the actual validity reaches a preset validity value; The data integration module is used to generate and execute data integration instructions according to the final feature subset, and the data integration instructions are used to integrate the feature subset with other energy data.

[0013] In a third aspect, the present application provides an intelligent terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the above-mentioned energy operation command and management method based on big data.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the above-mentioned energy operation command and management method based on big data.

[0015] In summary, compared with the prior art, the above technical solution has the following beneficial effects: (1) The energy operation command and management system and method based on big data described in this application can take into account the comprehensiveness of data acquisition and the scientific nature of feature selection. First, it emphasizes the comprehensive acquisition of energy data, which covers multiple data units and provides a rich information basis for subsequent analysis. By generating and executing feature selection instructions, it can accurately select feature subsets that have a significant impact on energy operations from massive data, ensuring the effectiveness and representativeness of the selected features; (2) The energy operation command and management system based on big data and the method thereof described in the present application can take into account both the rationality of the evaluation function and the clarity of the stopping criterion, use the evaluation function to quantitatively evaluate the feature subset, and through the preset database query and the stopping criterion related to the evaluation function, the method can terminate unnecessary searches in time during the evaluation process, thereby improving the efficiency of the algorithm and avoiding waste of resources; (3) The energy operation command and management system based on big data and the method described in this application can take into account the independence of the verification data set and the rationality of the preset effective value, emphasize the use of an independent verification data set to verify the actual effectiveness of the selected feature subset, and ensure the objectivity and reliability of the verification result. By comparing with the preset effective value, it is possible to scientifically judge whether the feature subset meets the actual needs, providing a strong basis for subsequent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of an energy operation command and management method based on big data in an embodiment of the present application; Figure 2 It is a schematic diagram of the process of generating and executing a data review instruction in an embodiment of the present application; Figure 3 It is a flowchart of identifying missing data values ​​that do not conform to a conventional pattern in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present application is further described in detail below in conjunction with all the accompanying drawings.

[0018] The present application embodiment discloses an energy operation command and management system based on big data and a method thereof, referring to Figure 1 , an energy operation command and management method based on big data includes: S101. Obtain energy data.

[0019] Specifically, the energy operation command and management system acquires energy data, which includes multiple data units and involves energy data collected from various energy systems (such as power systems, gas systems, thermal systems, etc.), thereby ensuring that subsequent analysis has sufficient data support and providing the possibility for extracting valuable information.

[0020] S102: Generate and execute feature selection instructions.

[0021] Specifically, the energy operation command and management system generates and executes a feature selection instruction based on the energy data, wherein the feature selection instruction is used to select a feature subset from multiple data units, wherein the feature subset includes a number of valid features. The energy operation command and management system selects a feature subset that has an important impact on energy operations based on the acquired energy data. This is conducive to reducing data dimensions, improving the efficiency of subsequent analysis, and ensuring that attention is paid to the factors that have the greatest impact on energy operations.

[0022] S103: Evaluate the feature subset using an evaluation function.

[0023] Specifically, the energy operation command and management system uses an evaluation function to evaluate the feature subset to obtain an evaluation function value. The energy operation command and management system measures the quality of the selected feature subset through an evaluation function, and the evaluation function may be constructed based on multiple factors such as the relevance, importance, and redundancy of the feature. This is conducive to ensuring that the selected feature subset is effective and optimized through quantitative evaluation, and provides a basis for subsequent decision-making.

[0024] S104: Query a stop criterion related to the evaluation function from a preset database.

[0025] Specifically, the energy operation command and management system queries the stopping criteria related to the evaluation function from a preset database. The user pre-sets the stopping criteria corresponding to the current energy data. The stopping criteria can be set as the evaluation function value reaching a certain threshold or the number of iterations reaching an upper limit, etc., to limit the number of times the evaluation function judges the current energy data.

[0026] S105. If the evaluation function value reaches the stopping criterion, the search is stopped.

[0027] Specifically, if the evaluation function value reaches the stopping criterion, the energy operation command and management system stops searching. In the evaluation process, by setting a certain stopping criterion, meaningless repeated calculations can be avoided. This can effectively improve the efficiency of the algorithm, avoid resource waste, and ensure satisfactory results within a reasonable time.

[0028] S106: Query a verification data set and a preset valid value corresponding to the feature subset from a preset database.

[0029] Specifically, the energy operation command and management system queries the verification data set and preset valid values ​​corresponding to the feature subset from a preset database, wherein the verification data set corresponds to the feature subset, and refers to the data set to which the current feature subset belongs, and the preset valid values ​​are used to verify the validity of the current feature subset, thereby determining whether the current feature subset meets the criteria for being included in the verification data set.

[0030] S107: Verify the actual validity of the feature subset.

[0031] Specifically, the energy operation command and management system verifies the actual validity of the feature subset based on the verification data set. The energy operation command and management system uses an independent verification data set to verify the effect of the selected feature subset in actual application. At the same time, it is compared with the preset effective value. This is conducive to ensuring that the selected feature subset is not only effective in theory, but also can achieve the expected effect in actual application, thereby improving the generalization ability of the model.

[0032] S108: If the actual validity reaches a preset validity value, output the final feature subset.

[0033] Specifically, if the actual effectiveness reaches the preset effective value, the final feature subset is output. If the actual effectiveness of the feature subset reaches the preset standard, the energy operation command and management system regards it as the final result output. This is conducive to providing reliable data support for subsequent energy operation decisions and helps to achieve more accurate energy management and scheduling.

[0034] S109: Generate and execute data integration instructions.

[0035] Specifically, the energy operation command and management system generates and executes a data integration instruction based on the final feature subset, and the data integration instruction is used to integrate the feature subset with other energy data. The energy operation command and management system integrates the final feature subset with other energy data to form a more comprehensive and valuable data set.

[0036] In another embodiment, S102 specifically includes the following sub-steps: Define the loss function: ; Among them: p represents the number of features, that is, the total number of independent variables; y i represents the target variable value of the i-th sample, that is, the dependent variable observation value; x ij represents the jth eigenvalue of the i-th sample, that is, the observed value of the independent variable; β j represents the p-dimensional coefficient vector, that is, the model parameters that need to be solved; n represents the number of samples, that is, the total number of observations; λ represents the regularization parameter, which is used to control the strength of regularization to prevent overfitting; Find the optimal coefficient vector β by minimizing the objective function; Due to the L1 regularization term The existence of will gradually reduce the coefficients of unimportant data features to 0 during the iteration process; In the final coefficient vector β, the data features with non-zero coefficients are considered important and used to form the feature subset; while the data features with coefficients compressed to 0 are considered redundant or unimportant and are eliminated from the feature subset.

[0037] Specifically, the energy operation command and management system can automatically select features during model training by defining a loss function containing an L1 regularization term, thereby reducing the complexity of the model and improving the generalization ability of the model. In machine learning, the objective function is usually a loss function plus a regularization term. By minimizing the objective function, the optimal model parameters (i.e., coefficient vector β) can be found. After iterative optimization, a set of optimal coefficient vectors β can be obtained, which reflects the degree of influence of each feature on the target variable.

[0038] During the iteration process, the L1 regularization term will gradually reduce the coefficients of unimportant data features to 0. This is because the L1 regularization term penalizes the absolute value of the coefficient, making the model more inclined to select a small number of important features and compress the coefficients of other features to 0. Through the effect of the L1 regularization term, sparse selection of features can be achieved, that is, redundant or unimportant features are automatically eliminated, and features that have a significant impact on the target variable are retained.

[0039] After the above steps, the data features with non-zero coefficients in the obtained coefficient vector β are considered important and used to form feature subsets. Data features whose coefficients are compressed to 0 are considered redundant or unimportant and are removed from the feature subset. The final feature subset not only reduces the data dimension, but also retains the most critical information, which helps to improve the interpretability and prediction accuracy of the model. At the same time, since redundant features are removed, the computational complexity and overfitting risk of the model can also be reduced.

[0040] Therefore, by defining a loss function containing an L1 regularization term and using an optimization algorithm to minimize the objective function, the purpose of sparse feature selection and automatic elimination of redundant features is achieved. This process not only improves the efficiency and generalization ability of the model, but also provides more accurate and effective data support for subsequent energy operation command and management.

[0041] The introduction of the regularization parameter λ provides a flexible means for controlling the complexity of the model. By adjusting the value of λ, the optimal balance between the model's fitting effect and generalization ability can be found to prevent overfitting. Compared with other feature selection methods, such as those based on statistical tests or model performance, the L1 regularization method is usually more computationally efficient. Especially when processing high-dimensional data, it can quickly converge to the optimal solution through optimization algorithms such as coordinate descent. The feature subsets obtained by the L1 regularization method are usually more interpretable. This is because the selected features (i.e., features with non-zero coefficients) directly reflect their significant relationship with the target variable.

[0042] After removing redundant features, the model can focus more on factors that have a substantial impact on the target variable, thereby improving the model's prediction accuracy and generalization ability. This is especially important for fields that require high-reliability predictions, such as energy operations. The reduction in the number of features means that the amount of data that needs to be processed during model training and prediction is reduced, thereby reducing computing and time costs. This is of great significance for large-scale data sets and application scenarios with high real-time requirements. Since the L1 regularization method can automatically identify important features, it has strong scalability. When the data set changes or new features need to be introduced, the model can adaptively adjust the feature subset to maintain the stability and reliability of its performance.

[0043] In another embodiment, the step of finding the optimal coefficient vector β by minimizing the objective function specifically includes the following sub-steps: Select the update dimension. In each iteration, select a dimension j (j=1, 2, 3...p) to update. Fix the coefficients of other dimensions β k (k≠j), calculate the residual r of the current dimension j; ; Where: r ij represents the residual of the i-th observation in the j-th dimension, that is, the prediction error after removing the influence of the current dimension; x ik represents the kth feature of the i-th observation; Calculate the part of the objective function in this dimension and replace the β in the objective function j Items are extracted separately and we get: ; Where: L j (β j ) represents the part of the objective function on dimension j, that is, only β is considered j The loss of time; x ij represents the jth feature of the i-th observation; Find the optimal solution for L j (β j ), calculate the β that minimizes it j ; When β j >0, for L j (β j ) and set it to 0, and we get: ; like , then the solution is valid; otherwise β j =0; When β j <0, the solution is: ; like , then the solution is valid; otherwise β j =0; When β j =0, if , then the solution is valid; After the iteration is completed, the final coefficient vector β is output, which is the optimal coefficient vector β that minimizes the objective function of LASSO.

[0044] Specifically, the energy operation command and management system selects the update dimension. In each iteration, the algorithm selects a dimension j to update, which means that the algorithm considers each coefficient β one by one. j , and tries to find the value that makes it optimal. By optimizing each coefficient one by one, the algorithm is able to gradually approach the global optimal solution. This method is simple and easy to implement, especially for high-dimensional data.

[0045] The energy operation command and management system fixes the coefficients of other dimensions, calculates the residual of the current dimension j, and then calculates the residual r of the current dimension j. ij When , the algorithm fixes the coefficients β of all other dimensions k (k=j), residual r ij It represents the prediction error of the model for the i-th observation after removing the influence of the current dimension j. This step is to simplify the problem to consider only one variable β j By fixing other coefficients, it is easier to find the optimal solution in the current dimension.

[0046] Then, the energy operation command and management system calculates the part of the objective function in this dimension and converts the β j The term is extracted separately by considering β j , we can see its impact on the objective function more directly, making it easier to find the optimal solution. j (β j ) and set it to 0, and then determine β according to the solution (whether it meets certain conditions). j The final value of , thereby finding the β that minimizes the objective function in the current dimension j Due to the L1 regularization term of LASSO, the solution may be 0, which helps to achieve feature selection. When the coefficients of all dimensions are optimized, the algorithm outputs the final coefficient vector β. The final coefficient vector β is the solution that minimizes the objective function of LASSO regression. Due to the effect of L1 regularization, this solution may contain many 0 values, thus achieving feature selection, that is, the model only relies on those features that contribute significantly to the prediction.

[0047] By adopting the coordinate descent strategy, only one dimension (coefficient β) is optimized in each iteration. j ) and fix the values ​​of all other dimensions. This strategy reduces the problem to a single-variable optimization problem at each step, which is easy to solve. For each selected dimension j, the method solves the problem by making L j (β j )The smallest β j The value is used to accurately update the coefficient of the dimension. At the same time, conditional judgment is used to determine whether to change β j Set to 0, which directly reflects the sparsity effect of L1 regularization.

[0048] Due to the effect of L1 regularization, in the iterative process, unimportant features (i.e., corresponding coefficients β j Smaller features) are gradually compressed to 0, thus achieving automatic feature selection. This method is more direct and effective than feature selection methods based on statistical tests or heuristic rules. Coordinate descent is usually more efficient than global optimization methods (such as gradient descent) in solving LASSO regression, especially when there are a large number of features. This is because it avoids updating all coefficients in each iteration, thereby reducing the amount of calculation and is easy to extend to large-scale data sets and high-dimensional feature spaces.

[0049] By automatically removing redundant features, the model becomes more concise and its interpretability is improved. Removing unimportant features helps reduce the overfitting of the model to the training data, thereby improving its prediction performance on unseen data. This is of great significance for improving the generalization and robustness of the model. Compared with the global optimization method, the coordinate descent method reduces the amount of calculation per iteration, thereby reducing the computational cost of model training and prediction. This is especially important for processing large-scale data sets and application scenarios with high real-time requirements.

[0050] Reference Figure 2 Further, after S102, as another implementation, the embodiment of the present application may further include the following steps: S201, generate and execute data audit instructions.

[0051] Specifically, the energy operation command and management system generates and executes data audit instructions based on the energy data. The data audit instructions are used to check energy data and retrieve erroneous data, missing data, and abnormal data. The energy operation command and management system conducts a comprehensive check on the original energy data in an automated manner. The focus of the audit is to identify and mark erroneous data (such as illogical values), missing data (unfilled or blank fields), and abnormal data (values ​​that are significantly different from most data points, which may be caused by measurement errors or equipment failures). This is conducive to ensuring the integrity and accuracy of the data, laying a solid foundation for subsequent analysis, and is conducive to improving data quality and reducing analysis deviations caused by data problems.

[0052] S202: If erroneous data, missing data and / or abnormal data are retrieved, a data processing instruction is generated and executed.

[0053] Specifically, if the energy operation command and management system retrieves erroneous data, missing data and / or abnormal data, it generates and executes data processing instructions, which are used to process erroneous data, missing data, and abnormal data, including processing abnormal values, filling missing values, deleting duplicate records, and correcting errors.

[0054] Once the energy operation command and management system identifies erroneous data, missing data and / or abnormal data, it needs to generate corresponding processing instructions, including but not limited to: Processing outliers: Correct or eliminate outliers through statistical methods (such as mean substitution, median substitution, quantile substitution, etc.) or domain knowledge. Fill missing values: Use reasonable methods (such as mean filling, mode filling, interpolation, etc.) to estimate and fill missing values. Delete duplicate records: Ensure that there are no duplicate data rows in the data set to avoid bias during analysis. Correct errors: Directly modify obvious erroneous data based on the audit results. It is conducive to further improving data quality and reducing noise and errors in the data. Make the data set more consistent and reliable, providing strong support for subsequent data analysis and modeling.

[0055] S203: Generate and execute data conversion instructions.

[0056] Specifically, the energy operation command and management system generates and executes data conversion instructions, which are used to perform data type conversion, date format processing, and normalize or standardize the data so that energy data of different magnitudes can be compared and analyzed. Data conversion is an important part of data preprocessing, and its main purpose is to convert raw data into a form suitable for analysis. This includes data type conversion (such as converting text to numeric type), date format processing (ensuring the consistency and accuracy of date data), and data normalization or standardization (enabling energy data of different magnitudes to be compared and analyzed on the same scale). It is conducive to improving the efficiency and accuracy of data analysis, avoiding errors caused by inconsistent data types or dimensional differences, making data easier to process and visualize, and facilitating subsequent analysis and decision making.

[0057] S204: Generate and execute data screening instructions.

[0058] Specifically, the energy operation command and management system generates and executes data screening instructions, which are used to screen the processed energy data. After the data has been processed as described above, the energy operation command and management system also needs to screen the data according to the analysis requirements. The screening work can be based on specific conditions or rules to extract a subset of data that is useful for analysis, for example, based on time range, region, energy type and other conditions. This is conducive to focusing on key data, reducing the complexity and amount of calculation of the analysis, improving the pertinence and effectiveness of the analysis, and making the analysis results more in line with actual needs.

[0059] In another embodiment, referring to Figure 3 , S201 specifically includes the following sub-steps: S201.1. Query data processing tools, preset audit rules and audit statistical rules from the preset database.

[0060] Specifically, the energy operation command and management system queries data processing tools, preset audit rules and audit statistical rules from the preset database. Data processing tools can include software or scripts for data cleaning, conversion and verification; preset audit rules define specific conditions that data should meet, such as data range, format, logical consistency, etc.; audit statistical rules focus on the statistical characteristics of data, such as mean, standard deviation, distribution pattern, etc. This is conducive to ensuring that subsequent steps can be operated based on accurate and comprehensive tools and rules, providing a solid foundation for data verification and processing.

[0061] S201.2. Verify the energy data according to the preset audit rules and audit statistical rules.

[0062] Specifically, the energy operation command and management system verifies the energy data according to the preset audit rules and audit statistical rules, and identifies abnormal data items that do not comply with the preset audit rules and / or audit statistical rules. The energy operation command and management system performs a comprehensive verification of the energy data, and applies a variety of methods such as logical judgment, condition matching, and statistical testing to identify abnormal items in the data. Abnormal data may include format errors, logical contradictions, values ​​outside a reasonable range, data points that do not conform to statistical characteristics, etc. This can effectively identify and isolate errors and anomalies in the data, providing a clear goal for subsequent data repair or elimination.

[0063] S201.3. Mark and store the abnormal data item and detailed information corresponding to the abnormal data item.

[0064] Specifically, the energy operation command and management system marks and stores the abnormal data items and the corresponding detailed information of the abnormal data items, and the detailed information includes field description, data type, location, error cause, data range, and timestamp. The energy operation command and management system marks the identified abnormal data items and records the relevant information in detail, such as field description (explaining the meaning of the data item), data type (such as numeric type, date type, etc.), location (specific location in the data set), error cause (why it is judged to be abnormal), data range (specific numerical range of abnormal values), and timestamp (the time when the data was recorded). This facilitates subsequent data tracking, analysis, and processing, and also provides feedback for the continuous improvement of data quality.

[0065] S201.4. Use statistical functions in data processing tools to detect missing values ​​in energy data.

[0066] Specifically, the energy operation command and management system detects missing values ​​in the energy data using the statistical functions in the data processing tool according to the data processing tool. The energy operation command and management system traverses the data set, checks whether each data item is filled or empty, and uses the statistical and analytical functions in the data processing tool to detect whether there are missing values ​​in the data set, which is conducive to accurately identifying the missing values ​​in the data set and providing a basis for subsequent data filling or deletion operations.

[0067] S201.5. Identify missing data values ​​that do not conform to regular patterns.

[0068] Specifically, the energy operation command and management system analyzes the distribution and pattern of energy data to identify missing data values ​​that do not conform to the normal pattern. After detecting missing values, the energy operation command and management system further analyzes the distribution and pattern of the data to identify those missing data that are not just randomly missing, but may be caused by specific reasons (such as equipment failure, data recording interruption, etc.) and do not conform to the normal pattern. This is conducive to a deeper understanding of the reasons and background of data missing, and helps to formulate more targeted data recovery or supplementation strategies.

[0069] S201.6. Mark and store the missing data value and detailed information corresponding to the missing data value.

[0070] Specifically, the energy operation command and management system marks and stores the missing data values ​​and the detailed information corresponding to the missing data values. The energy operation command and management system marks the missing data values ​​and records their detailed information, including the location of the missing values, possible causes, related data patterns or distribution characteristics, etc. This ensures that the missing data problem is fully recorded and tracked, providing necessary information support for subsequent data processing and analysis.

[0071] During the data review phase, the energy operation command and management system will first query the required data processing tools, audit rules, and audit statistical rules from the preset database. Then, the energy operation command and management system will conduct a comprehensive check on the energy data according to these rules to identify abnormal data items that do not comply with the rules. These abnormal data items may be caused by measurement errors, equipment failures, or data entry errors. Once abnormal data is identified, the system will mark and store these data items and their detailed information (such as field description, data type, location, error cause, data range, timestamp, etc.).

[0072] It helps ensure the accuracy and reliability of data and provides a high-quality data foundation for subsequent analysis. Through detailed abnormal records, it is convenient for subsequent data correction and tracking. It improves the automation of data processing and reduces the need for manual intervention.

[0073] Reference Figure 3 Further, after S201.6, as another implementation, the embodiment of the present application may further include the following steps: S201.7. Obtain key indicator data from a preset database.

[0074] Specifically, the energy operation command and management system obtains key indicator data from the preset database; in this step, the system retrieves the data indicators that are considered to be the most critical in the business or analysis process from the preset database. These indicators are usually of great significance to decision-making, performance evaluation or process monitoring. These key indicator data include but are not limited to key performance indicators (KPIs), business key success factors (KSFs), or professional data indicators in specific fields. It is helpful to clarify the priority of data processing and ensure that the data that is most important to business or analysis can be given priority attention and processing when resources are limited.

[0075] S201.8. If there is key indicator data in the missing data value, a warning instruction is generated and executed.

[0076] Specifically, if there is key indicator data in the data missing value, the energy operation command and management system generates and executes an early warning instruction, and the early warning instruction is used to push the data missing value and the detailed information corresponding to the data missing value to the user's smart terminal.

[0077] When it is determined that there are missing values ​​in the data set, and these missing values ​​contain key indicator data, the system will automatically generate an early warning instruction. The purpose of this early warning instruction is to promptly notify relevant personnel to pay attention to the missing data so that remedial measures can be taken quickly, such as re-collecting data, estimating missing values, or adjusting analysis strategies. It is conducive to ensuring that relevant personnel can quickly understand the situation of missing data and reduce analysis delays or decision-making errors caused by missing data. Since the early warning instruction targets the missing data of key indicators, it can attract the attention of relevant personnel and promote the rapid resolution of the problem. By providing detailed records of missing values ​​and their detailed information, the transparency and traceability of data processing are increased, which helps to improve data quality and the accuracy of analysis.

[0078] The execution of the early warning instruction is not just sending a simple notification, but pushing a comprehensive record of missing values ​​and their detailed information to the user's smart terminal (such as mobile phones, tablets, smart watches, etc.) in an easy-to-read and understand way. This push method should support multiple device types and operating systems to ensure that users can receive early warning information anytime and anywhere and take corresponding actions.

[0079] Users do not need to actively query the data status, they can learn about data missing at any time through smart terminals, which improves the convenience of information acquisition. Users can respond to warning information immediately, reducing decision delays or losses caused by information lag.

[0080] In another embodiment, S204 specifically includes the following sub-steps: The data unit is defined as feature f, and the relevance score of feature f is defined as: ; Where: f i represents the value of sample i on feature f; S ij Represents the corresponding value in the weight matrix S. S is a simulated expression of the data space structure, describing the distance between two samples of the same type (neighborhood). The greater the distance between two samples of the same type (neighborhood), the greater the corresponding weight, and vice versa. Var(f) represents the variance of feature f. Query a preset score threshold from a preset database; Among all data units, select the preferred data units whose relevance scores reach the preset score threshold.

[0081] Specifically, the energy operation command and management system defines the data unit as feature f and calculates a relevance score for each feature. This score takes into account the feature value f i , the weight matrix S between samples (simulating the data space structure, especially the distance relationship between samples of the same type), and the variance Var(f) of the feature. The design of the weight matrix S makes the distance between samples of the same type (or neighboring samples) a factor affecting the importance of the feature. The larger the distance, the larger the corresponding weight, which to a certain extent reflects the ability of the feature to distinguish different samples. By comparing the relevance score of each feature with the preset score threshold, the data units (i.e., features) that are more important to the analysis task can be screened out.

[0082] By quantifying the importance of features, the energy operation command and management system can more accurately select data units that contribute to analysis, reduce data redundancy, and focus more on features that have a significant impact on the results, thereby improving the efficiency and accuracy of data analysis. It helps to discover potential data patterns or rules and provide support for subsequent decision-making.

[0083] Based on the above method, the embodiment of the present application also discloses an energy operation command and management system based on big data. An energy operation command and management system based on big data includes: An energy data acquisition module, the energy data acquisition module is used to acquire energy data, the energy data includes a plurality of data units; A feature selection module, the feature selection module is used to generate and execute a feature selection instruction according to the energy data, the feature selection instruction is used to select a feature subset from a plurality of data units, the feature subset includes a plurality of valid features; A feature subset evaluation module, the feature subset evaluation module is used to evaluate the feature subset using an evaluation function to obtain an evaluation function value; A stop criterion query module, the stop criterion query module is used to query the stop criterion related to the evaluation function from a preset database; Search module, the search module is used to stop searching if the evaluation function value reaches the stopping criterion; A verification data set query module, the verification data set query module is used to query the verification data set and preset valid values ​​corresponding to the feature subset from a preset database; A feature verification module, the feature verification module is used to verify the actual validity of the feature subset based on the verification data set; The feature output module is used to output the final feature subset if the actual validity reaches a preset validity value.

[0084] The data integration module is used to generate and execute data integration instructions according to the final feature subset, and the data integration instructions are used to integrate the feature subset with other energy data.

[0085] An embodiment of the present application also discloses an intelligent terminal, which includes a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor such as the above-mentioned energy operation command and management method based on big data.

[0086] The embodiment of the present application also discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program that can be loaded by a processor and executes the above-mentioned energy operation command and management method based on big data. The computer-readable storage medium includes, for example, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in this field can still combine, add, delete or make other adjustments to the features in the various embodiments of the present invention according to the circumstances without conflict, without making creative work, so as to obtain different other technical solutions that do not deviate from the concept of the present invention in essence, and these technical solutions also belong to the scope of protection of the present invention.

Claims

1. A method for energy operation command and management based on big data, characterized in that: The following steps are involved: Acquiring energy data, wherein the energy data includes a plurality of data units; Generate and execute a feature selection instruction according to the energy data, wherein the feature selection instruction is used to select a feature subset from a plurality of data units, wherein the feature subset includes a plurality of valid features; Using an evaluation function to evaluate the feature subset, to obtain an evaluation function value; Querying a stop criterion related to the evaluation function from a preset database; If the evaluation function value reaches the stopping criterion, the search stops; Querying a verification data set and a preset valid value corresponding to the feature subset from a preset database; Verifying the actual validity of the feature subset according to the verification data set; If the actual validity reaches the preset validity value, the final feature subset is output; According to the final feature subset, a data integration instruction is generated and executed, wherein the data integration instruction is used to integrate the feature subset with other energy data.

2. According to the energy operation command and management method based on big data according to claim 1, it is characterized in that: The steps of generating and executing feature selection instructions specifically include: Define the loss function: ; Among them: p represents the number of features, that is, the total number of independent variables; y i represents the target variable value of the i-th sample, that is, the dependent variable observation value; x ij represents the jth eigenvalue of the i-th sample, that is, the observed value of the independent variable; β j represents the p-dimensional coefficient vector, that is, the model parameters that need to be solved; n represents the number of samples, that is, the total number of observations; λ represents the regularization parameter, which is used to control the strength of regularization to prevent overfitting; Find the optimal coefficient vector β by minimizing the objective function; Due to the L1 regularization term The existence of will gradually reduce the coefficients of unimportant data features to 0 during the iteration process; In the final coefficient vector β, the data features with non-zero coefficients are considered important and used to form the feature subset; while the data features with coefficients compressed to 0 are considered redundant or unimportant and are eliminated from the feature subset.

3. According to the energy operation command and management method based on big data according to claim 2, it is characterized in that: The step of finding the optimal coefficient vector β by minimizing the objective function specifically includes: Select the update dimension. In each iteration, select a dimension j (j=1, 2, 3...p) to update. Fix the coefficients of other dimensions β k (k≠j), calculate the residual r of the current dimension j; ; Where: r ij represents the residual of the i-th observation in the j-th dimension, that is, the prediction error after removing the influence of the current dimension; x ik represents the kth feature of the i-th observation; Calculate the part of the objective function in this dimension and replace the β in the objective function j Items are extracted separately and we get: ; Where: L j (β j ) represents the part of the objective function on dimension j, that is, only β is considered j The loss of time; x ij represents the jth feature of the i-th observation; Find the optimal solution for L j (β j ), calculate the β that minimizes it j ; When β j >0, for L j (β j ) and set it to 0, and we get: ; like , then the solution is valid; otherwise β j =0; When β j <0, the solution is: ; like , then the solution is valid; otherwise β j =0; When β j =0, if , then the solution is valid; After the iteration is completed, the final coefficient vector β is output, which is the optimal coefficient vector β that minimizes the objective function of LASSO.

4. According to the energy operation command and management method based on big data according to claim 1, it is characterized in that: Before the step of generating and executing the feature selection instruction, the method further includes: Generate and execute data review instructions based on the energy data, wherein the data review instructions are used to check the energy data and retrieve erroneous data, missing data, and abnormal data; If erroneous data, missing data and / or abnormal data are retrieved, data processing instructions are generated and executed, and the data processing instructions are used to process the erroneous data, missing data, and abnormal data, including processing abnormal values, filling missing values, deleting duplicate records, and correcting errors; Generate and execute data conversion instructions, wherein the data conversion instructions are used to convert data types, process date formats, and normalize or standardize data so that energy data of different magnitudes can be compared and analyzed; Generate and execute a data screening instruction, wherein the data screening instruction is used to screen the processed energy data.

5. According to the energy operation command and management method based on big data according to claim 4, it is characterized in that: The steps of generating and executing the data screening instruction specifically include: The data unit is defined as feature f, and the relevance score of feature f is defined as: ; Where: f i represents the value of sample i on feature f; S ij Represents the corresponding value in the weight matrix S. S is a simulated expression of the data space structure, describing the distance between two samples of the same type (neighborhood). The greater the distance between two samples of the same type (neighborhood), the greater the corresponding weight, and vice versa. Var(f) represents the variance of feature f. Query a preset score threshold from a preset database; Among all data units, select the preferred data units whose relevance scores reach the preset score threshold.

6. The energy operation command and management method based on big data according to claim 4 is characterized in that: The steps of generating and executing the data review instruction specifically include: Query data processing tools, preset audit rules and audit statistics rules from the preset database; Verify the energy data according to the preset audit rules and audit statistical rules, and identify abnormal data items that do not comply with the preset audit rules and / or audit statistical rules; Marking and storing the abnormal data item and detailed information corresponding to the abnormal data item, wherein the detailed information includes field description, data type, location, error cause, data range, and timestamp; According to the data processing tool, using statistical functions in the data processing tool to detect missing data values ​​in the energy data; By analyzing the distribution and pattern of energy data, missing values ​​that do not conform to the regular pattern are identified; The missing data value and detailed information corresponding to the missing data value are marked and stored.

7. The energy operation command and management method based on big data according to claim 6 is characterized in that: After the step of marking and storing the missing data value and the detailed information corresponding to the missing data value, the method further includes: Obtain key indicator data from the preset database; If there is key indicator data in the data missing value, an early warning instruction is generated and executed, and the early warning instruction is used to push the data missing value and detailed information corresponding to the data missing value to the user's smart terminal.

8. An energy operation command and management system based on big data, characterized in that: include: An energy data acquisition module, used to acquire energy data, wherein the energy data includes a plurality of data units; A feature selection module, used for generating and executing a feature selection instruction according to the energy data, wherein the feature selection instruction is used for selecting a feature subset from a plurality of data units, wherein the feature subset includes a plurality of valid features; A feature subset evaluation module, used to evaluate the feature subset using an evaluation function to obtain an evaluation function value; A stop criterion query module, used to query a stop criterion related to the evaluation function from a preset database; A search module, used to stop searching if the evaluation function value reaches a stopping criterion; A verification data set query module, used to query a verification data set and a preset valid value corresponding to the feature subset from a preset database; A feature verification module, used to verify the actual validity of the feature subset based on the verification data set; A feature output module, used for outputting a final feature subset if the actual validity reaches a preset validity value; The data integration module is used to generate and execute data integration instructions according to the final feature subset, and the data integration instructions are used to integrate the feature subset with other energy data.

Citation Information

Patent Citations

  • Feature selection integration method and system containing large-scale wind power integration

    CN115374874A

  • Central air conditioning system energy efficiency optimization method based on data mining

    CN116384700A

  • Credit evaluation method and device based on EBM model, electronic equipment and storage medium

    CN117172910A

  • Data mining model construction method and system based on machine learning

    CN118332034A

  • Power big data processing method based on data clustering algorithm

    CN118503730A

Cited By

  • Artificial intelligence-based on-duty efficiency comprehensive evaluation method

    CN119740990A

  • An integrated evaluation method for duty efficiency based on artificial intelligence

    CN119740990B