Energy data warehouse-based configuration and processing method and device and related medium

Abstract

This invention discloses a configuration and processing method, apparatus, and related media based on an energy data warehouse. The method includes acquiring incremental energy data and location information, configuring effective location data; performing threshold configuration rule judgment to output a cleaning result table; wherein the cleaning result table includes environmental data and electricity meter data; performing outlier processing on the electricity meter data and the environmental data respectively to output first processed data and second processed data, ultimately overwriting to obtain a final cleaning result table; performing data processing based on the final cleaning result table to obtain categorized storage data and statistical storage data respectively; receiving query instructions and returning an energy information list. This invention processes the final cleaning result table to obtain categorized storage data and statistical storage data, and then uses query instructions to obtain an energy information list for managing the energy data warehouse. This solves the problem of data anomalies in the energy data warehouse and greatly improves management efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a configuration and processing method, apparatus and related media based on an energy data warehouse. Background Technology

[0002] Most existing data warehouse systems are not optimized for the specific needs of the energy industry. This results in insufficient data support for optimizing energy consumption, thus impacting energy management and energy conservation efforts. While many solutions exist in current technologies, none can meet the specific requirements of energy data warehouses. The energy industry frequently faces data anomaly issues, which traditional energy data warehouses cannot effectively address. Existing technology (patent number: CN112015724A) uses fixed values ​​to determine monthly electricity consumption and fixed rates to determine rated current, but this is not suitable for processing other electrical data, significantly reducing management efficiency. Summary of the Invention

[0003] The embodiments of the present invention provide a configuration and processing method, apparatus and related media based on an energy data warehouse, which aims to solve the problems of energy data warehouses being unable to effectively solve data anomalies and low management efficiency in the prior art.

[0004] In a first aspect, embodiments of the present invention provide a configuration and processing method based on an energy data warehouse, comprising:

[0005] Acquire incremental energy data and extract location information from the dataset in the configuration system;

[0006] The incremental energy data is configured using the location information to obtain effective location data;

[0007] The effective point data is judged according to the threshold configuration rules. If it meets the threshold configuration rules, a cleaning result table is output. If it does not meet the threshold configuration rules, the effective point data is processed for outliers and then a cleaning result table is output. The cleaning result table includes environmental data and electricity meter data.

[0008] The meter data is processed for outliers to output continuous first processed data.

[0009] Outlier processing is performed on the environmental data to output continuous second processed data;

[0010] The first and second processed data are time-series processed respectively, and then overwritten into the cleaning result table to obtain the final cleaning result table.

[0011] Based on the final cleaning result table, data classification and statistical analysis are performed to obtain classified and stored data and statistical stored data, respectively.

[0012] Based on the categorized and statistical stored data, the system receives query commands and returns a list of energy information.

[0013] Secondly, embodiments of the present invention provide a configuration and processing apparatus based on an energy data warehouse, comprising:

[0014] The data acquisition unit is used to acquire incremental energy data and extract the location information of the dataset in the configuration system;

[0015] A data configuration unit is used to configure the incremental energy data using the location information to obtain effective location data;

[0016] The data judgment unit is used to judge the effective point data according to the threshold configuration rules. If it meets the threshold configuration rules, it outputs a cleaning result table; if it does not meet the threshold configuration rules, it performs outlier processing on the effective point data and outputs a cleaning result table. The cleaning result table includes environmental data and electricity meter data.

[0017] The first processing unit is used to process the abnormal values ​​of the meter data so as to output continuous first processed data.

[0018] The second processing unit is used to process outliers in the environmental data in order to output continuous second processed data.

[0019] The data overwriting unit is used to perform time-series processing on the first processed data and the second processed data respectively, and overwrite them into the cleaning result table to obtain the final cleaning result table.

[0020] The data storage unit is used to classify and statistically analyze the data based on the final cleaning result table, and to obtain classified storage data and statistical storage data, respectively.

[0021] The data output unit is used to receive query instructions based on the classified and statistical stored data and return a list of energy information.

[0022] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the configuration and processing method based on an energy data warehouse as described in the first aspect.

[0023] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the configuration and processing method based on an energy data warehouse of the first aspect.

[0024] This invention provides a configuration and processing method based on an energy data warehouse, including acquiring incremental energy data and location information, configuring effective location data; performing threshold configuration rule judgment, and finally outputting a cleaning result table; wherein the cleaning result table includes environmental data and electricity meter data; performing outlier processing on the electricity meter data and the environmental data respectively to output first processed data and second processed data, and finally overwriting to obtain a final cleaning result table; performing data processing based on the final cleaning result table to obtain categorized storage data and statistical storage data respectively; receiving query instructions and returning an energy information list. This invention processes the final cleaning result table to obtain categorized storage data and statistical storage data, and then uses query instructions to obtain an energy information list for managing the energy data warehouse. This solves the problem of data anomalies in the energy data warehouse and greatly improves management efficiency.

[0025] The present invention also provides a configuration and processing device, computer equipment and storage medium based on an energy data warehouse, which have the same beneficial effects as described above. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a configuration and processing method based on an energy data warehouse, provided as an embodiment of the present invention;

[0028] Figure 2 This is a schematic block diagram of a configuration and processing device based on an energy data warehouse, provided for an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] Please see below. Figure 1 , Figure 1 The flowchart of a configuration and processing method based on an energy data warehouse provided in an embodiment of the present invention specifically includes steps S101 to S108.

[0034] S101. Obtain incremental energy data and extract the location information of the dataset in the configuration system;

[0035] S102. Configure the incremental energy data using the location information to obtain effective location data;

[0036] S103. The effective point data is judged according to the threshold configuration rule. If it meets the threshold configuration rule, the cleaning result table is output. If it does not meet the threshold configuration rule, the effective point data is processed for outliers and then the cleaning result table is output. The cleaning result table includes environmental data and electricity meter data.

[0037] S104. Perform outlier processing on the meter data to output continuous first processed data;

[0038] S105. Perform outlier processing on the environmental data to output continuous second processed data;

[0039] S106. The first processed data and the second processed data are time-series converted respectively, and written to the cleaning result table to obtain the final cleaning result table.

[0040] S107. Based on the final cleaning result table, perform data classification and data statistics respectively to obtain classified storage data and statistical storage data respectively;

[0041] S108. Receive a query instruction based on the classified storage data and statistical storage data, and return an energy information list.

[0042] In step S101, acquiring incremental energy data refers to collecting real-time or historical data related to energy production, conversion, and utilization. This data can include information such as power plant capacity, output of wind or solar power generation devices, and changes in energy supply and demand. To acquire this data, various data acquisition technologies can be used, such as sensors, smart meters, and monitoring equipment, to monitor and record the status and performance of the energy system in real time. A common approach is to use Internet of Things (IoT) technology, deploying sensors in key locations such as power plants, transmission lines, and energy conversion equipment. These sensors can measure various parameters, such as current, voltage, temperature, and humidity, to obtain detailed data about the energy system. This data can be collected, stored, and processed through a cloud platform to provide real-time energy monitoring and analysis.

[0043] Specifically, extracting the location information of a dataset from a configuration system refers to obtaining the location information of the dataset from the configuration system. A configuration system is a system used to manage and organize datasets, which can include databases, data warehouses, or data management platforms. In a configuration system, datasets are typically organized as tables or files, and each data point is described using specific fields or identifiers. To extract the location information of a dataset, it is first necessary to understand the structure of the configuration system and how the dataset is organized. This information can be obtained by querying the configuration system's metadata (e.g., table structure, field descriptions). The metadata describes the name, data type, and meaning of each field in the dataset, as well as their location within the dataset. After understanding the structure and location information of the dataset, data can be extracted from the configuration system using a programming language or query language (such as SQL). By specifying specific fields, conditions, or identifiers, the desired data points can be selected from the dataset and exported or further processed.

[0044] Furthermore, with the rapid development of renewable energy, the demand for monitoring and managing renewable energy systems is increasing. Advanced data acquisition and analysis technologies can be used to monitor the output of devices such as solar photovoltaic panels and wind turbines in real time and optimize energy production. The vast amounts of data generated in the energy sector require efficient processing and analysis; big data technologies, such as distributed computing, machine learning, and artificial intelligence, can be applied to energy data to extract useful insights and perform predictive analysis. Based on energy data and location information, energy management systems can be developed to help monitor energy consumption in real time, optimize energy utilization, and implement energy-saving measures, providing functions such as energy reports, alarm notifications, and energy optimization suggestions. Advances in IoT and sensor technologies are making energy system monitoring and data acquisition more intelligent and automated. By connecting sensors to energy equipment and infrastructure, real-time data acquisition and remote monitoring can be achieved, improving the efficiency and reliability of energy systems.

[0045] In one embodiment, step S101 includes:

[0046] Read the project relationship table and the point identification table of the dataset in the configuration system respectively;

[0047] The project relationship table and the location identification table are merged to obtain the location information.

[0048] In this embodiment, the project relationship table and the point identification table are two important data tables in the configuration system. The project relationship table (eco_data_block) records the relationship between project numbers and dataset numbers. The steps for reading the project relationship table are as follows:

[0049] a. Determine the location and naming of the project relationship table: Locate the table that stores project relationship information in the configuration system; it usually has a specific table name or identifier.

[0050] b. Use an appropriate query language (such as SQL) to execute a query to retrieve the contents of the project relationship table. For example, you can use a SELECT statement to select all fields and records in the project relationship table.

[0051] c. Read the query results and parse the fields in the project relationship table.

[0052] The site identifier table (eco_block_device_ref) records the relationship between dataset numbers and all site identifiers required for energy consumption algorithms or analysis. The steps to read the site identifier table are as follows:

[0053] a. Locate the table in the configuration system that stores location identification information; it usually has a specific table name or identifier.

[0054] b. Use an appropriate query language to execute the query to retrieve the contents of the location identification table. For example, you can use a SELECT statement to select all fields and records in the location identification table.

[0055] c. Read the query results and parse the fields in the point identification table.

[0056] Finally, having obtained the data from the project relationship table and the location identification table, we can merge the data to obtain the location information. The data merging steps are as follows:

[0057] a. The project relationship table and the location identification table can be linked by using a linking operation (such as JOIN);

[0058] b. Merge the records in the project relationship table and the location identification table to create a new data table containing information on project relationships and corresponding locations;

[0059] c. By reading the merged data table and parsing its fields, complete location information can be obtained, including project name, location name, location coordinates, etc.

[0060] In addition, depending on the complexity of the configuration system and the size of the dataset, reading and merging data may require more advanced technologies and tools; for example, data integration platforms or ETL (Extract, Transform, Load) tools can be used to handle the reading and merging of large-scale datasets. These tools provide more powerful data processing and transformation capabilities and can support complex data integration needs.

[0061] In step S102, location information can be associated with corresponding energy data to obtain valid location data. Through the data configuration process, energy data with configured location information, i.e., valid location data, is obtained and can be used for subsequent analysis, visualization, or other applications. Of course, in addition to configuring historical incremental energy data, location information can also be configured with real-time energy data. This is very useful for applications that require real-time monitoring and control of energy systems, providing immediate location data updates. By using location information to configure incremental energy data and obtaining valid location data, the performance of energy systems can be better understood and analyzed, energy utilization optimized, and energy management decisions supported. This provides an important data foundation for achieving sustainable energy development and energy efficiency.

[0062] In one embodiment, step S102 includes:

[0063] Write the incremental energy data into the database to obtain the project ID and the location ID;

[0064] The project ID and location ID are merged with the location information to obtain the location configuration result; wherein, the location configuration result includes valid location data and invalid location data.

[0065] In this embodiment, incremental energy data is written to a database to obtain project IDs and location IDs. Then, the data is merged to obtain the location configuration results, including valid and invalid location data. First, the incremental energy data is imported into the database using database management tools or interfaces provided by programming languages, such as the SQL INSERT statement. The data is inserted into the database tables according to the corresponding field structure and data type to ensure accuracy and completeness. In the database, the project IDs and location IDs are obtained by querying the incremental energy data table. These IDs can be used as association keys to associate and merge with the location information. Using an appropriate query language (such as SQL), a query is executed to select the project ID and location ID fields and extract the corresponding values ​​from the incremental energy data table. Finally, the location information and incremental energy data are merged using the project IDs and location IDs.

[0066] Specifically, location data refers to incremental energy data that can be successfully associated and merged with location information. This data contains a project ID, location ID, and corresponding energy data fields. Valid location data is usable for subsequent analysis and application. Invalid location data refers to incremental energy data for which matching location information could not be found during the association and merging process. This may be due to incomplete location information, data quality issues, or configuration errors. Invalid location data may require further processing and investigation to resolve configuration problems or correct data errors.

[0067] In step S103, for valid location data, threshold configuration rules can be applied for judgment, and a cleaning result table is output based on the results of the rules. If the data meets the threshold configuration rules, it is output directly; if it does not meet the rules, outlier processing is required before outputting the cleaning result table. The cleaning result table includes environmental data and electricity meter data. The following is a detailed explanation of the threshold configuration rule judgment: Appropriate threshold configuration rules are defined according to specific application requirements. These rules can be based on the upper and lower limits of the data, the rate of change, trends, or other relevant conditions. The threshold configuration rules are applied to the valid location data for judgment, each data point is compared with the threshold defined in the rules, and the processing method is determined based on the judgment results. The data is categorized based on its type, such as environmental data and electricity meter data. Data that meets the threshold configuration rules is directly output to the cleaning result table; this data is considered normal and requires no further processing. Data that does not meet the threshold configuration rules needs to undergo outlier handling before being output to the cleaning result table. Outlier handling can employ techniques such as interpolation, smoothing, or deletion to correct or remove abnormal data. The environmental data and electricity meter data that meet the threshold rules and have undergone outlier handling are merged to generate the cleaning result table. The cleaning result table can include the original data, the processed data, and other relevant information for subsequent analysis and application.

[0068] Generally speaking, valid data points correspond to one or more threshold rules. The values ​​in the incremental data are compared with the threshold rule requirements, and if they meet the requirements, they are considered valid. For example, the threshold rule for point 1 (such as outdoor temperature) is >=-10 and <=50. Then, the outdoor temperature data in the incremental data (such as values ​​of 26.5 and 57.8) will be associated with two rules: rule 1 is >=-10 and rule 2 is <=50. 26.5 meets the requirements of each corresponding threshold rule, while 67.8 only meets the requirements of rule 1 and does not meet the requirements of rule 2.

[0069] Specifically, when processing data that does not conform to the threshold configuration rules, different outlier handling methods can be applied. For example, interpolation methods can be used to fill in missing values, filtering techniques can be used to smooth data, or statistical methods can be used to detect and remove outlier data points. Threshold configuration rules can be dynamically adjusted according to different time periods, seasonal changes, or specific events, which can better adapt to data changes and characteristics, improving the accuracy and adaptability of data cleaning. When generating the cleaning result table, data quality indicators and reports can be added to describe the quality status of the data. This can include assessments of data completeness, accuracy, and consistency to help users understand the credibility and usability of the data. In addition to basic threshold configuration and outlier handling, more advanced cleaning methods can be applied, such as anomaly detection algorithms, machine learning models, or time series analysis. These methods can more accurately identify and process outlier data, improving the effectiveness and efficiency of data cleaning. By applying threshold configuration rules and performing outlier handling, cleaned environmental data and meter data can be obtained. These cleaning results help improve data quality, accuracy, and usability, providing a more reliable foundation for subsequent analysis, modeling, and decision-making.

[0070] In one embodiment, step S103 includes:

[0071] When the output rule type of the valid point data is determined to be filtering, the outliers of the valid point data are invalidated.

[0072] When the output rule type of the valid point data is determined to be the default value, the abnormal values ​​of the valid point data are changed to the default value, and the cleaning result table is output.

[0073] When the output rule type of the valid point data is determined to be algorithm assignment, the abnormal values ​​of the valid point data are nulled, and the cleaning result table is output.

[0074] In this embodiment, if the output rule type is filtering, it means that outliers will be discarded, i.e., deleted from the cleaning result table. Data that meets the threshold configuration rules is directly output to the cleaning result table; data that does not meet the threshold configuration rules has its outliers discarded and removed from the cleaning result table. If the output rule type is default value, it means that outliers will be changed to predefined default values ​​and then output to the cleaning result table. Data that meets the threshold configuration rules is directly output to the cleaning result table; data that does not meet the threshold configuration rules has its outliers replaced with predefined default values ​​and then output to the cleaning result table. This way, all data points are retained in the cleaning result table, but outliers are replaced with default values. If the output rule type is algorithm assignment, it means that outliers will be set to null and then output to the cleaning result table. Data that meets the threshold configuration rules is directly output to the cleaning result table; data that does not meet the threshold configuration rules has its outliers set to null and then output to the cleaning result table. This way, all data points are retained in the cleaning result table, but outliers are set to null values.

[0075] In addition, alerts can be generated for processed outliers, allowing relevant personnel to be promptly aware of data anomalies and take appropriate measures. Besides fixed output rule types, dynamic output rules can be implemented based on actual conditions and needs. For example, output rules can be dynamically adjusted according to changes in time, data type, or other parameters to better address different data situations and business requirements. Further outlier analysis can be performed during outlier processing to understand the causes and underlying data patterns. This helps identify potential problems or system failures and allows for appropriate corrective measures. Besides outputting to a cleansing results table, the cleansing results can be output in different formats, such as reports, charts, and visualization dashboards, to meet diverse user needs and usage scenarios.

[0076] In step S104, outliers in the meter data can be directly deleted, and the remaining data will maintain continuity after deletion. Alternatively, interpolation methods can be used to fill out outliers. Interpolation methods can infer reasonable values ​​for outliers based on the values ​​of adjacent data points. Common interpolation methods include linear interpolation and spline interpolation. The data after interpolation maintains continuity. For cases with many outliers or where outliers have a significant impact on data continuity, smoothing methods can be used to process outliers. Smoothing methods smooth noise and outliers in the data by filtering or averaging the data, thereby obtaining the first processed data with better continuity.

[0077] In one embodiment, step S104 includes:

[0078] Determine whether there are missing or duplicate meter data in the meter data. If there are missing or duplicate meter data, perform missing or duplicate processing and then perform outlier detection. If there are no missing or duplicate meter data, directly perform outlier detection.

[0079] If outliers are identified, the jump data is removed and the point data is filled with data, and then the outliers are cleared. If no outliers are identified, the outliers are cleared directly. The clearing process includes: extracting the first N and last N points of the point data to form a reasonable data range, clearing the point data outside the reasonable data range, and filling with data.

[0080] Based on the result of the emptying process, the first processed data is output continuously.

[0081] In this embodiment, checking for missing meter data points can be done by comparing the continuity of the data points; if some data points have intervals in the time series, then missing data points may exist. Checking for duplicate data points can be done by comparing their uniqueness; duplicate data points indicate that the same point appeared multiple times at the same time. If missing data points exist, they need to be filled in. This can be done using interpolation or other appropriate methods to infer the missing data point value based on preceding and following data points. If duplicate data points exist, deduplication is required, which removes duplicate data points to ensure that each point has only one value at a specific time. Outlier detection is performed on the processed data points. This can be done by comparing the deviation of data points from a reasonable data range using threshold configuration rules or other methods. If outliers exist, they need to be processed. Specifically, this can involve removing jump data and filling in the data points, or clearing data points outside the reasonable range and filling in the data points. For outliers, data points exceeding the reasonable data range are set to null or empty values. After this process, the nulled data points can be filled in using the trends and values ​​of the previous N and next N data points. Filling methods can include interpolation, averaging, linear regression, etc. Based on the results of the nulling and data filling, a first-processed dataset with good continuity is output. This first-processed dataset can be used as input data for subsequent energy analysis, load forecasting, and other tasks.

[0082] In addition to interpolation and deduplication, other methods can be applied to handle missing or duplicate data points. These include data imputation methods based on statistical laws or models, or time series analysis to infer the values ​​of missing or duplicate data. Besides threshold configuration rules, other advanced outlier detection methods, such as anomaly detection algorithms and machine learning models, can be used to more accurately identify and process outliers. During processing, establishing a data quality monitoring system and providing timely feedback on the results helps identify potential data quality issues and ensures that the processed data meets requirements. By comprehensively handling missing or duplicate data points, outlier detection, nulling, and data imputation, a first-stage processed data with good continuity can be obtained. This data can then provide a reliable foundation for subsequent tasks such as energy analysis, load forecasting, and energy efficiency assessment.

[0083] In step S105, outlier processing is performed on the environmental data to output continuous second-processed data. Outliers can be directly deleted from the environmental data, and the remaining data points will maintain continuity after deletion. For cases with few outliers, interpolation methods can be used to fill in the outliers. Interpolation methods can infer reasonable values ​​for outliers based on the values ​​of adjacent data points. Common interpolation methods include linear interpolation and spline interpolation. The data after interpolation processing maintains continuity. For cases with many outliers or outliers that significantly affect data continuity, smoothing methods can be used to process outliers. Smoothing methods smooth noise and outliers in the data by filtering or averaging, thereby obtaining second-processed data with better continuity.

[0084] In one embodiment, step S105 includes:

[0085] Determine whether there are missing or duplicate point data in the environmental data. If there are missing or duplicate point data, perform missing or duplicate processing and then perform outlier detection. If there are no missing or duplicate point data, directly perform outlier detection.

[0086] If outliers are detected, the jump data is removed and the point data is filled with data, and then the continuous second processed data is output; if no outliers are detected, the continuous second processed data is output directly.

[0087] In this embodiment, it is determined whether there are missing or duplicate location data in the environmental data. If there are missing or duplicate data, appropriate processing is required before outlier detection. If there are no missing or duplicate data, outlier detection can be performed directly, as follows:

[0088] a. Check if there are missing data points in the environmental data. This can be determined by comparing the continuity of the data points. If there are intervals in the time series of some data points, there may be missing data points.

[0089] b. Check if there are duplicate location data in the environmental data. You can determine if there are duplicates by comparing the uniqueness of the location data. If there are duplicate location data, it means that the same location appeared multiple times at the same time.

[0090] c. If there are missing data points, they need to be filled in. Filling in the missing data points can be done by interpolation or other appropriate methods, based on the data points before and after the missing data points.

[0091] d. If duplicate data points exist, deduplication is required. Deduplication can delete duplicate data points to ensure that each data point has only one value at a specific time.

[0092] Outlier identification is as follows:

[0093] a. To identify outliers in the processed environmental data, the deviation of data points from a reasonable data range can be compared using threshold configuration rules or other methods to determine whether outliers exist.

[0094] b. If outliers exist, they need to be processed. Specifically, the processing method can be to remove the jump data and fill the data of the point. Outlier processing can also be to empty the point data that exceeds the reasonable data range and then fill the data.

[0095] Furthermore, for the outlier-processed data, methods and techniques for handling outliers ensure the continuity of the output second-processed data, maintaining consistency with the original data in both value and trend. Continuity assessment and verification of the second-processed data can be performed using methods such as data visualization and statistical analysis. For missing and duplicate data points, in addition to interpolation and deduplication, other methods can be applied, such as data imputation methods based on statistical laws or models, or time series analysis to infer the values ​​of missing or duplicate data. Besides threshold configuration rules, other advanced outlier detection methods, such as anomaly detection algorithms and machine learning models, can be applied. These methods can more accurately identify outliers and handle them accordingly. During the processing, a data quality monitoring system should be established to provide timely feedback on the processing results. This helps to identify potential data quality problems and ensure that the processed data meets the requirements. By comprehensively handling missing or duplicate data points, outlier detection, and continuity verification, second-processed environmental data with good continuity can be obtained, thus providing a reliable foundation for subsequent energy analysis, modeling, and decision-making.

[0096] In step S106, the first processed data is sorted chronologically to ensure the data is arranged in chronological order. If the data is already arranged chronologically, no further sorting is needed. Similarly, the second processed data is sorted chronologically to ensure data continuity and correct time series. The time-series-processed first processed data is written into the corresponding environmental data / electricity meter data column in the cleaning result table; similarly, the time-series-processed second processed data is written into the corresponding environmental data / electricity meter data column in the cleaning result table. If necessary, the cleaning result table may also include other information columns related to the project, location, or data to provide more complete data analysis and processing. The final cleaning result table will contain the processed and time-series-processed first and second processed data.

[0097] In addition, before overwriting the data into the cleaned results table, data quality checks and anomaly handling can be performed to ensure the quality and consistency of the first and second processed data, and to repair or mark outliers to guarantee the data quality of the final cleaned results table. For data security and traceability, the cleaned results table should be backed up regularly, and an appropriate version management mechanism should be established to allow for backtracking to a specific version of the cleaned results when needed, ensuring data reliability and reproducibility. The final cleaned results table can be obtained by time-series sequencing of the first and second processed data and then overwriting it.

[0098] In step S107, the data in the cleaning result table is categorized and stored. A data storage structure indexed by project ID and location ID can be created to facilitate rapid data access and querying based on project and location. Database tables, file systems, or other storage mechanisms can be used to organize and store the categorized data as needed. Indexes are added to the categorized data to improve retrieval efficiency. Data for specific projects and locations can be quickly located and retrieved based on the project ID and location ID indexes. Categorized storage facilitates subsequent data analysis, visualization, and report generation. Statistical analysis is performed on the data in the cleaning result table as needed to obtain statistically stored data. For example, average, maximum, minimum, and sum values ​​for each project and location can be calculated. The statistical results can be stored in database tables, data files, or other data storage structures as required. Choose appropriate statistical methods and techniques based on actual needs. Commonly used statistical methods include mean, median, standard deviation, percentiles, etc. Statistical calculations can be performed using statistical software, programming languages, or data analysis tools. Use stored statistical data for data visualization and report generation to better understand and communicate the statistical characteristics and trends of the data. Use charts, graphs, tables, and other forms to display statistical results to support data analysis and decision-making processes.

[0099] In addition to categorizing and storing data based on project ID and location ID, advanced data classification and tagging techniques can be applied. Data can be categorized and tagged with finer granularity based on its characteristics, attributes, or other criteria to meet the needs of data analysis at different levels and dimensions. When performing data statistics, more advanced statistical analysis methods and models can be applied, such as regression analysis, time series analysis, and cluster analysis. These methods can reveal relationships, trends, and patterns between data, providing deeper data insights and decision support. If real-time data classification and statistics are required, real-time data processing and streaming computing technologies can be considered. These technologies can process data streams and update categorized and stored data and statistical results in real time to respond promptly to and analyze changing data. Through data classification and statistics, the final cleaned result table can be organized and stored by project and location, while simultaneously obtaining the statistical characteristics of each project and location. Such categorized and statistically stored data provides convenience and a foundation for further data analysis, trend analysis, and decision-making.

[0100] In step S108, a query interface or system is established to receive query instructions sent by the user. The query instructions may include parameters such as project ID, location ID, time range, and statistical indicators, used to specify query conditions and requirements. The parameters in the query instructions are parsed, such as extracting the project ID and location ID, determining the time range for the query, and specifying the statistical operations to be performed. Based on the received query instructions, the corresponding energy data is retrieved from the categorized storage data. Indexes or other query techniques can be used to quickly locate and retrieve data for specified projects and locations. Based on the statistical requirements in the query instructions, the corresponding energy data statistical results are queried and retrieved from the statistical storage data. Depending on the statistical requirements, corresponding statistical calculation operations are performed, and the calculated statistical results are returned. Based on the query results, the obtained energy data is organized and formatted into an energy information list. The formatted energy information list is returned to the user as the query result, which the user can receive and use through an interface, application, or other means.

[0101] Furthermore, in addition to basic queries, more advanced query functions can be supported, such as flexible querying and filtering by time range, data type, and statistical indicators to meet diverse user query needs. For large-scale data and complex query scenarios, query optimization techniques and performance improvement measures can be adopted, such as using indexes, data partitioning, and caching to accelerate query operations and improve query response speed and efficiency. The energy information list can be visualized, for example, presenting trends and changes in energy data through charts, graphs, and maps; simultaneously, users can interact with the query results, such as zooming, filtering, and exporting, to meet users' needs for in-depth data analysis and use. By receiving query commands, querying and retrieving data, and returning the query results as an energy information list, flexible querying and use of categorized and statistically stored data can be achieved.

[0102] In summary, this invention utilizes algorithms and data cleaning techniques to accurately remove outliers and fill in missing values, thereby improving the accuracy of energy data. It can dynamically determine the reasonable data range for each period based on historical records, thus reducing the cost of energy data processing. By automating and optimizing the handling of energy data anomalies, it reduces the manpower and time costs associated with manually processing abnormal data. Accurate and reliable energy data enables better monitoring of energy consumption and the identification of energy-saving opportunities, thereby improving energy management efficiency and reducing costs. It can also promptly detect anomalies in energy use and potential energy-saving opportunities, helping decision-makers make more rational energy management decisions. Furthermore, by providing more accurate and reliable energy data, it allows for better evaluation of the effectiveness of energy-saving algorithms, which helps improve the level of energy-saving algorithms, promotes sustainable energy development, and drives the implementation of more effective energy-saving measures. Through these advantages, this invention can provide higher-quality energy data, helping the energy industry achieve more effective energy management and electricity conservation, contributing to sustainable energy development.

[0103] In one embodiment, the configuration and processing method based on an energy data warehouse further includes:

[0104] By using the temperature and electricity data queried from the energy information list, and combining them with historical temperature and energy consumption relationships, the energy efficiency ratio is obtained.

[0105] The COP value is obtained by querying the chilled water flow rate, chilled water supply temperature, and chilled water return temperature using the energy information list and combining them with the power of the chiller.

[0106] By comparing historical electricity consumption data with real-time electricity consumption data using the energy information list, electricity consumption trend information can be obtained.

[0107] In this embodiment, based on the energy information list, the following analysis and calculations can be performed:

[0108] Query temperature and electricity data to obtain the energy efficiency ratio: Extract the required temperature data from the energy information list, which can be indoor temperature, ambient temperature, or other relevant temperature data; extract the required electricity data from the energy information list, which can be total electricity consumption or electricity consumption of specific equipment; establish a relationship model between temperature and energy consumption based on historical data or known relationships, for example, by using regression analysis to determine the linear or nonlinear relationship between temperature and energy consumption; substitute the obtained temperature data into the model based on the temperature and energy consumption relationship model to calculate the energy efficiency ratio, which reflects the efficiency of energy consumption relative to temperature changes.

[0109] By querying the chilled water flow rate, chilled water supply temperature, and chilled water return temperature, and combining this with the chiller's power, the COP value can be obtained.

[0110] Data such as chilled water flow rate, chilled water supply temperature, and chilled water return temperature are extracted from the energy information list; power data of the chiller is also extracted from the energy information list; using the chilled water flow rate, chilled water supply temperature, chilled water return temperature, and chiller power data, the COP (Coefficient of Performance) value of the refrigeration system can be calculated. The COP value is an indicator of the refrigeration system performance, reflecting the cooling effect per unit power input.

[0111] By comparing historical and real-time electricity consumption data, electricity consumption trend information can be obtained:

[0112] Retrieve historical electricity consumption data from the energy information list, which can be hourly, daily, or monthly; query real-time electricity consumption data: obtain current real-time electricity consumption data; compare historical and real-time data: compare historical and real-time electricity consumption data to analyze their differences and trends. Calculate indicators such as electricity consumption growth rate and percentage change to obtain trend information on electricity consumption.

[0113] Furthermore, when calculating energy efficiency ratios and COP values, more advanced data analysis methods and models, such as time series analysis and machine learning, can be employed. These methods can more accurately establish the relationships between temperature and energy consumption, and between chilled water flow and temperature, yielding more precise results. In addition to querying energy information lists, data such as temperature, electricity consumption, chilled water flow, and chiller power can be monitored in real time. Real-time monitoring allows for timely detection and response to changes in energy consumption and efficiency, further optimizing energy management and energy-saving measures. Energy efficiency ratios, COP values, and electricity consumption trends can be displayed and communicated through data visualization and report generation. Charts, trend analyses, and report summaries help to more intuitively understand and apply these energy data analysis results. By querying and analyzing the energy information list, temperature and electricity data can be obtained. Combined with historical temperature and energy consumption relationships, chilled water flow rate, chilled water supply temperature, chilled water return temperature, and chiller power, the energy efficiency ratio and COP value can be obtained. By comparing historical and real-time electricity consumption data, trend information on electricity consumption can be obtained. These analytical results can be used for energy efficiency assessment, optimization of energy-saving measures, and energy management decisions.

[0114] Combination Figure 2 As shown, Figure 2 A schematic block diagram of a configuration and processing device based on an energy data warehouse provided in an embodiment of the present invention. The configuration and processing device 200 based on the energy data warehouse includes:

[0115] The data acquisition unit 201 is used to acquire incremental energy data and extract the location information of the dataset in the configuration system;

[0116] Data configuration unit 202 is used to configure the incremental energy data using the location information to obtain effective location data;

[0117] The data judgment unit 203 is used to judge the effective point data according to the threshold configuration rule. If it meets the threshold configuration rule, it outputs a cleaning result table; if it does not meet the threshold configuration rule, it performs outlier processing on the effective point data and outputs a cleaning result table. The cleaning result table includes environmental data and electricity meter data.

[0118] The first processing unit 204 is used to process the abnormal values ​​of the electricity meter data so as to output continuous first processed data.

[0119] The second processing unit 205 is used to perform outlier processing on the environmental data in order to output continuous second processed data.

[0120] The data overwriting unit 206 is used to perform time-series conversion on the first processed data and the second processed data respectively, and overwrite them into the cleaning result table to obtain the final cleaning result table.

[0121] Data storage unit 207 is used to perform data classification and data statistics based on the final cleaning result table, and obtain classified storage data and statistical storage data respectively;

[0122] The data output unit 208 is used to receive query instructions based on the classified storage data and statistical storage data, and return an energy information list.

[0123] In this embodiment, the data acquisition unit 201 acquires incremental energy data and extracts location information from the dataset in the configuration system; the data configuration unit 202 uses the location information to configure the incremental energy data to obtain valid location data; the data judgment unit 203 performs threshold configuration rule judgment on the valid location data; if it meets the threshold configuration rule, it outputs a cleaning result table; if it does not meet the threshold configuration rule, it performs outlier processing on the valid location data and outputs a cleaning result table; wherein, the cleaning result table includes environmental data and electricity meter data; the first processing unit 204 processes the electricity meter data. Outlier processing is performed to output continuous first processed data; second processing unit 205 performs outlier processing on the environmental data to output continuous second processed data; data overwriting unit 206 performs time-series processing on the first and second processed data respectively and overwrites them into the cleaning result table to obtain the final cleaning result table; data storage unit 207 performs data classification and data statistics based on the final cleaning result table to obtain classified storage data and statistical storage data respectively; data output unit 208 receives query instructions based on the classified storage data and statistical storage data and returns an energy information list.

[0124] In one embodiment, the data acquisition unit 201 includes:

[0125] The reading unit is used to read the project relationship table and the point identification table of the dataset in the configuration system, respectively.

[0126] The merging unit is used to merge the project relationship table and the location identification table to obtain the location information.

[0127] In one embodiment, the data configuration unit 202 includes:

[0128] The writing unit is used to write the incremental energy data into the database to obtain the project ID and the location ID;

[0129] The configuration unit is used to merge the project ID and the location ID with the location information to obtain the location configuration result; wherein, the location configuration result includes valid location data and invalid location data.

[0130] In one embodiment, the data determination unit 203 includes:

[0131] The filtering unit is used to invalidate the outlier values ​​of the valid point data when the output rule type of the valid point data is filtering.

[0132] The default unit is used to change the abnormal values ​​of the valid point data to the default value when the output rule type of the valid point data is the default value, and output the cleaning result table.

[0133] The assignment unit is used to determine that when the output rule type of the valid point data is algorithm assignment, it performs numerical nulling on the abnormal values ​​of the valid point data and outputs the cleaning result table.

[0134] In one embodiment, the first processing unit 204 includes:

[0135] The repair unit is used to determine whether there are missing or duplicate point data in the meter data. If there are missing or duplicate point data, the corresponding missing or duplicate processing is performed, and then the outlier judgment is performed; if there are no missing or duplicate point data, the outlier judgment is performed directly.

[0136] The emptying unit is used to, if it is determined that there are outliers, remove jump data and fill the point data with data, and then empty the outliers; if it is determined that there are no outliers, it directly emptys the outliers; wherein, the emptying process includes: extracting the first N and last N points of the point data to form a reasonable data range, emptying the point data outside the reasonable data range, and filling the data.

[0137] The meter unit is used to output continuous first processed data based on the result of the emptying process.

[0138] In one embodiment, the second processing unit 205 includes:

[0139] The judgment unit is used to determine whether there are missing or duplicate point data in the environmental data. If there are missing or duplicate point data, the corresponding missing or duplicate processing is performed, and then outlier judgment is performed; if there are no missing or duplicate point data, outlier judgment is performed directly.

[0140] The environment unit is used to, if an outlier is detected, remove the jump data, fill in the data points, and then output the continuous second processed data; if no outlier is detected, it directly outputs the continuous second processed data.

[0141] In one embodiment, the configuration and processing apparatus based on the energy data warehouse further includes:

[0142] The energy consumption data unit is used to obtain the energy efficiency ratio value by using the temperature data and power data queried from the energy information list and combining them with the historical temperature and energy consumption relationship.

[0143] The power data unit is used to query the chilled water flow rate, chilled water supply temperature, and chilled water return temperature using the energy information list, and to obtain the COP value by combining it with the power of the chiller unit;

[0144] The trend data unit is used to compare historical electricity consumption data and real-time electricity consumption data using the energy information list to obtain electricity consumption trend information.

[0145] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0146] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0147] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0149] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A configuration and processing method based on an energy data warehouse, characterized in that, include: Acquire incremental energy data and extract location information from the dataset in the configuration system; The incremental energy data is configured using the location information to obtain effective location data; The effective point data is judged according to the threshold configuration rules. If it meets the threshold configuration rules, the cleaning result table is output. If the threshold configuration rules are not met, the valid point data will be processed for outliers and a cleaning result table will be output; wherein, the cleaning result table includes environmental data and electricity meter data; The meter data is processed for outliers to output continuous first processed data. Outlier processing is performed on the environmental data to output continuous second processed data; The first and second processed data are time-series processed respectively, and then overwritten into the cleaning result table to obtain the final cleaning result table. Based on the final cleaning result table, data classification and statistical analysis are performed to obtain classified and stored data and statistical stored data, respectively. Based on the categorized and statistical stored data, a query command is received, and a list of energy information is returned. The step of configuring the incremental energy data using the location information to obtain valid location data includes: writing the incremental energy data into a database to obtain a project ID and a location ID; merging the project ID and location ID with the location information to obtain a location configuration result; wherein the location configuration result includes valid location data and invalid location data.

2. The configuration and processing method based on an energy data warehouse according to claim 1, characterized in that, The acquisition of incremental energy data and extraction of location information from the dataset in the configuration system includes: Read the project relationship table and the point identification table of the dataset in the configuration system respectively; The project relationship table and the location identification table are merged to obtain the location information.

3. The configuration and processing method based on an energy data warehouse according to claim 1, characterized in that, If the threshold configuration rules are not met, the effective point data will undergo outlier processing to output a cleaning result table, including: When the output rule type of the valid point data is determined to be filtering, the outliers of the valid point data are invalidated. When the output rule type of the valid point data is determined to be the default value, the abnormal values ​​of the valid point data are changed to the default value, and the cleaning result table is output. When the output rule type of the valid point data is determined to be algorithm assignment, the abnormal values ​​of the valid point data are nulled, and the cleaning result table is output.

4. The configuration and processing method based on an energy data warehouse according to claim 1, characterized in that, The process of handling outliers in the meter data to output continuous first processed data includes: Determine whether there are missing or duplicate meter data in the meter data. If there are missing or duplicate meter data, perform missing or duplicate processing and then perform outlier detection. If there are no missing or duplicate meter data, directly perform outlier detection. If outliers are identified, the jump data is removed and the point data is filled with data, and then the outliers are cleared. If no outliers are identified, the outliers are cleared directly. The clearing process includes: extracting the first N and last N points of the point data to form a reasonable data range, clearing the point data outside the reasonable data range, and filling with data. Based on the result of the emptying process, the first processed data is output continuously.

5. The configuration and processing method based on an energy data warehouse according to claim 1, characterized in that, The outlier processing of the environmental data to output continuous second processed data includes: Determine whether there are missing or duplicate point data in the environmental data. If there are missing or duplicate point data, perform missing or duplicate processing and then perform outlier detection. If there are no missing or duplicate point data, directly perform outlier detection. If outliers are detected, the jump data is removed and the point data is filled with data, and then the continuous second processed data is output; if no outliers are detected, the continuous second processed data is output directly.

6. The configuration and processing method based on an energy data warehouse according to claim 1, characterized in that, Also includes: By using the temperature and electricity data queried from the energy information list, and combining them with historical temperature and energy consumption relationships, the energy efficiency ratio is obtained. The COP value is obtained by querying the chilled water flow rate, chilled water supply temperature, and chilled water return temperature using the energy information list and combining them with the power of the chiller. By comparing historical electricity consumption data with real-time electricity consumption data using the energy information list, electricity consumption trend information can be obtained.

7. A configuration and processing device based on an energy data warehouse, characterized in that, include: The data acquisition unit is used to acquire incremental energy data and extract the location information of the dataset in the configuration system; A data configuration unit is used to configure the incremental energy data using the location information to obtain effective location data; The data judgment unit is used to judge the threshold configuration rules of the effective point data. If the threshold configuration rules are met, the cleaning result table is output. If the threshold configuration rules are not met, the valid point data will be processed for outliers and a cleaning result table will be output; wherein, the cleaning result table includes environmental data and electricity meter data; The first processing unit is used to process the abnormal values ​​of the meter data so as to output continuous first processed data. The second processing unit is used to process outliers in the environmental data in order to output continuous second processed data. The data overwriting unit is used to perform time-series processing on the first processed data and the second processed data respectively, and overwrite them into the cleaning result table to obtain the final cleaning result table. The data storage unit is used to classify and statistically analyze the data based on the final cleaning result table, and to obtain classified storage data and statistical storage data, respectively. The data output unit is used to receive query instructions based on the classified and statistical stored data and return a list of energy information. The data configuration unit is specifically used to write the incremental energy data into the database to obtain the project ID and the location ID; to merge the project ID and the location ID with the location information to obtain the location configuration result; wherein, the location configuration result includes valid location data and invalid location data.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the configuration and processing method based on an energy data warehouse as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the configuration and processing method based on an energy data warehouse as described in any one of claims 1 to 6.

Images