Energy data preprocessing method, system and equipment

Abstract

The invention relates to the technical field of data processing, and discloses an energy data preprocessing method, system and device. The method comprises the following steps: acquiring energy data of each equipment node, and sending the energy data to message middleware; calling message middleware to perform consumption processing on the energy data, determining consumption data, and storing the consumption data as cache data in a cache table of a database; the cache data are read and preprocessed, extracted service data are obtained, and preprocessing comprises cleaning, conversion and classification; and combining the business data according to the time dimension and the business dimension, so that the business data of the same type are stored in the same data table of the database. On one hand, query, analysis and report generation are facilitated; on the other hand, through ordered organization and storage of the data, the data access efficiency is improved, and the business expandability is also improved; and the business trend can be understood more easily, potential problems can be identified, and corresponding solutions can be formulated.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to an energy data preprocessing method, system and device. Background Art

[0002] Energy data plays an extremely important role in the energy industry and social and economic fields. Through the collection and analysis of energy data, on the one hand, it is possible to monitor the energy consumption of equipment in real time, discover energy waste problems, and thus optimize energy use strategies and improve energy efficiency; on the other hand, the energy data provided by energy consumption in different regions and industries provides a scientific basis for resource allocation. Energy data based on monitoring is conducive to the rational layout of energy infrastructure by governments and enterprises, and improves the safety and reliability of energy supply.

[0003] However, in related technologies, the amount of collected device data is huge, and the storage and management of device data are not flexible and intelligent enough, resulting in limited data analysis and visualization functions, poor application performance, high difficulty, and complex technical architecture for data processing, which is not conducive to the analysis and display of energy data. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the present application discloses an energy data preprocessing method, system and device, which are used to solve the problems of poor data storage and scalability caused by business coupling, poor system scalability and difficulty in change when preprocessing energy data.

[0005] The first aspect of the present application discloses an energy data preprocessing method, comprising: obtaining energy data of each device node and sending the energy data to a message middleware; calling the message middleware to perform consumption processing on the energy data, determining consumption data, and storing the consumption data as cache data in a cache table of a database; reading the cache data and preprocessing it to obtain extracted business data, wherein the preprocessing includes cleaning, conversion and classification; combining the business data according to the time dimension and the business dimension so that the business data of the same type are stored in the same data table of the database.

[0006] In certain embodiments of the first aspect of the present application, before combining the business data according to the time dimension and the business dimension, it also includes: obtaining configuration parameters and parsing the configuration parameters to determine the computing cycle, energy consumption unit, energy type, time dimension and business dimension of the computing node to be configured; configuring the business data according to the computing cycle, the energy consumption unit, the energy type, the time dimension and the business dimension, and generating a computing node for processing the business data; extracting the cached data, sending the cached data to the computing node for execution, and generating summary data.

[0007] In certain embodiments of the first aspect of the present application, the business data is configured according to the configuration parameters, and a computing node for processing the business data is generated, including: if it is monitored that the time dimension corresponding to the cache data of any of the device nodes is greater than the preset time dimension, the time dimension is split into multiple computing cycles according to the preset computing cycle; each computing cycle creates a computing node according to the business dimension, and the number of computing nodes is determined by the product of the number of computing cycles and the type of the business dimension; a computing task is configured for each computing node, and input parameters are allocated to the computing node according to the computing task, so that each computing node executes the computing tasks in parallel.

[0008] In certain embodiments of the first aspect of the present application, a computing task is configured for each of the computing nodes, including: calling a service interface to configure an extension point to expand the business logic of the custom business; decomposing the business logic of the custom business to decompose the custom business into multiple computing subtasks; configuring the computing subtasks to each of the computing nodes through a custom class loader, and loading each of the computing subtasks to generate business data.

[0009] In certain embodiments of the first aspect of the present application, the message middleware is called to perform consumption processing on the energy data to determine the consumption data, including: extracting the type and processing requirements of the energy data and storing them in a database; judging whether the energy data is to be consumed based on the processing requirements of the energy data; if it is determined that the energy data is to be consumed, calling the queue of the preset topic of the message middleware for consumption processing based on the type of the energy data, and storing the generated consumption data in a pending table; for the pending messages in the pending table, consumption processing is performed in real-time consumption processing mode or timed consumption processing mode, and after the consumption processing of the consumption data is completed, the pending messages in the pending table are cleared.

[0010] In certain embodiments of the first aspect of the present application, the business data is combined according to the time dimension and the business dimension so that the business data of the same type are stored in the same data table of the database, including: using a business summary interface to collect each business data; dividing each business data according to the business dimension, so that the business data of each type of business dimension is stored in the database using a data table; sorting the business data of the same business type from front to back according to the time dimension, so that the business data from the same device node is stored in the same subtable in the database.

[0011] In certain embodiments of the first aspect of the present application, the cache data is read and preprocessed to obtain the extracted business data, including: performing distributed data cleaning on the cache to obtain cleaned data, the data cleaning includes missing values, deduplication, error correction and outliers; performing distributed data conversion on the cleaned data to determine it as converted data in a preset format, the data conversion includes at least one of the following: data standardization, data encoding, data aggregation and data reconstruction; performing distributed classification processing on the converted data to obtain the extracted business data, the classification processing includes data segmentation and / or business classification.

[0012] In certain embodiments of the first aspect of the present application, after the business data of the same type are stored in the same data table of the database, it also includes: in response to a business request instruction, accessing the database to determine associated business data, the associated business data including multiple sub-data; determining the standard deviation of the associated business data and the residual error of each sub-data, and determining the random error distribution range based on a preset multiple and the standard deviation; judging whether the residual error is outside the random error distribution range; when the residual error is outside the random error distribution range, eliminating the sub-data corresponding to the residual error to obtain the target business data corresponding to the business request instruction.

[0013] The second aspect of the present application discloses an energy data preprocessing system, including: an acquisition module, used to acquire energy data of each device node and send the energy data to a message middleware; a consumption processing module, used to call the message middleware to perform consumption processing on the energy data, determine the consumption data, and store the consumption data as cache data in a cache table of a database; a preprocessing module, used to read the cache data and perform preprocessing to obtain extracted business data, wherein the preprocessing includes cleaning, conversion and classification; a data storage module, used to combine the business data according to the time dimension and the business dimension so that the business data of the same type are stored in the same data table of the database.

[0014] The third aspect of the present application discloses an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0015] The beneficial effects of the present application are as follows: by calling the message middleware to process the consumption of energy data, the consumption data is determined and stored in the cache table of the database, so that the efficiency and response speed of data processing can be improved; by performing pre-processing operations such as cleaning, conversion and classification on the cache data, not only the data quality and consistency are improved, but also the accuracy and reliability of the data are improved; the business data is combined according to the time dimension and the business dimension, and stored in the same data table of the database, on the one hand, it is conducive to query, analysis and report generation; on the other hand, through the orderly organization and storage of data, not only the efficiency of data access is improved, but also the business scalability is improved; it can also make it easier to understand business trends, identify potential problems, and formulate corresponding solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an application environment diagram of an energy data preprocessing system in an embodiment of the present application;

[0017] Figure 2 It is a flowchart of an energy data preprocessing method in an embodiment of the present application;

[0018] Figure 3 This is an overall architecture diagram of an energy data preprocessing system in an embodiment of the present application;

[0019] Figure 4 This is a complete flow chart of an energy data preprocessing method in an embodiment of the present application;

[0020] Figure 5 This is a flowchart of a node generation interface in an energy data preprocessing method in an embodiment of the present application;

[0021] Figure 6 This is a flow chart of a data aggregation interface in an energy data preprocessing method in an embodiment of the present application;

[0022] Figure 7 It is a structural schematic diagram of an energy data preprocessing system in an embodiment of the present application;

[0023] Figure 8 It is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and sub-samples in the embodiments can be combined with each other without conflict.

[0025] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0026] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0027] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0028] Unless otherwise stated, the term "plurality" means two or more.

[0029] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0030] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0031] Combination Figure 1 As shown, an implementation environment diagram of an energy data preprocessing system provided by an embodiment of the present disclosure includes a user end and a server end, wherein the user end and the server end are connected to a network and communicate with each other. In this application, the user end is the front end and the server end is the back end.

[0032] It should be understood that the energy data preprocessing system is applied to the user side or the server side. In some embodiments, the user side can be at least one of a computer device, a desktop device, a smart phone, and a tablet device; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and it can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms; the software can be an application program of the energy data preprocessing method, etc., but is not limited to the above forms.

[0033] Combination Figure 2 As shown, the embodiment of the present disclosure provides a flow chart of an energy data preprocessing method, including:

[0034] Step S201, obtaining energy data of each device node, and sending the energy data to the message middleware;

[0035] It should be understood that each device node includes an electricity meter, a water meter, a heat meter, a gas meter, etc. For example, if the device node is an electricity meter, energy data of electric energy, such as current, voltage, power, etc., are collected in real time; if the device node is a water meter, energy data of water energy, such as flow rate, flow velocity and other parameters, are collected in real time.

[0036] Message middleware uses an efficient and reliable message passing mechanism to exchange data independently of the platform, and is integrated into distributed systems based on data communication. For example, it can solve the data transmission needs of distributed systems, enable synchronous and asynchronous connections between clients and servers, and transmit and forward messages at any time.

[0037] The message middleware includes RabbitMQ, Kafka, ActiveMQ, RocketMQ, etc., which will not be discussed here.

[0038] Step S202, calling the message middleware to perform consumption processing on the energy data, determining consumption data, and storing the consumption data as cache data in a cache table of a database;

[0039] Optionally, in some embodiments, calling the message middleware to perform consumption processing on the energy data to determine the consumption data includes:

[0040] Extracting the type and processing requirements of the energy data and storing them in a database;

[0041] Determining whether the energy data is to be consumed according to the processing requirements of the energy data;

[0042] If it is determined that the energy data is to be consumed, the queue of the preset topic of the message middleware is called for consumption processing according to the type of the energy data, and the generated consumption data is stored in the pending table; for the pending messages in the pending table, consumption processing is performed in real-time consumption processing mode or timed consumption processing mode, and after the consumption processing of the consumption data is completed, the pending messages in the pending table are cleared.

[0043] For example, energy data is extracted from various energy collection devices, and the types of energy data are identified and classified according to the format, content or source characteristics of the energy data, such as electrical energy, water energy, thermal energy, etc., and the classified energy data is stored in a database, for example, using different tables or fields to distinguish different types of data.

[0044] According to business needs or data characteristics, the processing requirements of each type of energy data are analyzed, including but not limited to real-time analysis, historical statistics, report generation, etc., and the processing requirements of each energy data or data batch are marked in the database for subsequent processing; according to the processing requirements of the data, determine whether consumption processing (such as data cleaning, conversion, analysis, etc.) is required. If it is determined that consumption processing is required, the queue of the preset topic of the message middleware is called according to the type of energy data. The preset topic is set according to the data type or processing requirements, and the energy data is sent to the corresponding queue, waiting for the consumer to consume and process; the data generated during the consumption processing (such as processing results, intermediate status, etc.) is stored in the pending table for subsequent processing or query.

[0045] Select real-time consumption processing mode or scheduled consumption processing mode according to business needs. Real-time consumption, for data that needs to be processed immediately, adopts real-time consumption mode to ensure the timeliness and accuracy of data processing. Scheduled consumption, for data with high non-real-time requirements, you can set scheduled tasks for consumption processing to optimize the use of system resources. According to the selected consumption mode, the data in the processing table is consumed. The processing process includes but is not limited to data cleaning, conversion, analysis, aggregation and other operations. After the consumption data processing is completed, the processed messages are cleared from the processing table to avoid repeated processing or occupying unnecessary storage space.

[0046] Real-time monitoring of each link in the data processing process, including data extraction, storage, consumption processing, etc., to ensure smooth and unimpeded processes; based on the monitoring results, optimize system performance, including adjusting the configuration of the message middleware, optimizing database queries, and improving consumption processing efficiency. For exceptions or errors that occur during the processing process, timely processing and recording are carried out to ensure the stability and reliability of the system. Through the above methods, energy data can be effectively processed, and consumption processing is carried out according to its processing requirements, and finally the processing results are stored in the database for subsequent use.

[0047] In this embodiment, by consuming the collected device data with low latency, a larger data throughput can be supported under the same resource conditions.

[0048] Step S203, reading the cached data and performing preprocessing to obtain extracted business data, wherein the preprocessing includes cleaning, conversion and classification;

[0049] Optionally, in some embodiments, reading the cache data and preprocessing it to obtain the extracted business data includes:

[0050] Performing distributed data cleaning on the cache to obtain cleaned data, wherein the data cleaning includes missing values, deduplication, error correction and outlier correction;

[0051] For example, dealing with missing values, identifying and dealing with missing values ​​in the data, by filling (such as using the mean, median), deleting, or interpolating. Removing duplicate records, checking and deleting or merging duplicate records in the data to avoid bias in subsequent analysis. Correcting erroneous data, checking and correcting obvious errors in the data, such as format errors, logical errors, or typographical errors. Dealing with outliers, identifying and dealing with outliers in the data (also called outliers), which are caused by measurement errors, data entry errors, or other reasons, by deleting, replacing, or converting to normal values.

[0052] Performing distributed data conversion on the cleaned data to determine the converted data in a preset format, wherein the data conversion includes at least one of the following: data standardization, data encoding, data aggregation, and data reconstruction;

[0053] For example, data standardization and normalization, by scaling the data to have the same scale or distribution, so that it is fairer and more accurate when compared or calculated. Data encoding, converting categorical data (such as text data) into numerical data for mathematical operations or analysis. This is achieved through methods such as label encoding and one-hot encoding. Data aggregation, summarizing or merging data according to one or more fields to generate higher-level data summaries or statistical information. Data reconstruction, adjusting the data structure, such as converting from wide format to long format (or vice versa), to suit specific analysis needs.

[0054] The converted data is subjected to distributed classification processing to obtain extracted business data, wherein the classification processing includes data segmentation and / or business classification.

[0055] For example, feature extraction extracts meaningful features from raw data that can reflect the inherent laws or business logic of the data. Data segmentation divides continuous data into different intervals or segments to better analyze the distribution or trend of the data. Business logic classification divides data into different categories or groups based on business logic or rules for business analysis or decision making.

[0056] In this embodiment, the data is cleaned. Due to the complexity of the business site, some obviously abnormal data is collected during the data collection process. Therefore, the obviously abnormal data is cleaned and the data that does not meet the processing standards is filtered; the data is converted. The data collected by different data collection platforms have their own special formats and names. At this time, in order to facilitate the use of data by third parties, the data needs to be converted to a unified format. It is only used to clean and convert data; the converted data is stored. In order to improve consumption performance, the data is temporarily stored in the warehouse for subsequent aggregation.

[0057] Step S204: combining the business data according to the time dimension and the business dimension, so that the business data of the same type are stored in the same data table of the database.

[0058] Specifically, the business data are collected using a business summary interface; the business data are divided according to business dimensions, so that the business data of each business dimension is stored in a database using a data table; the business data of the same business type are sorted from front to back according to the time dimension, so that the business data from the same device node are stored in the same subtable in the database.

[0059] First, connect with the system or service interface that provides business data to ensure that the latest business data can be obtained in real time or regularly through the business summary interface, and receive business data returned by the interface, including but not limited to multiple business types, device node information, timestamps, etc. Secondly, according to business needs and data characteristics, identify different business dimensions, such as order processing, inventory changes, user behavior, etc., design one or more data tables for each business dimension, ensure that the data of each type of business dimension can be clearly stored in one or more tables, divide the received business data according to the business dimension, and store it in the corresponding data table. Each data table should contain necessary fields, such as business ID, timestamp, device node ID, business details, etc.

[0060] In addition, the computing performance of the system is improved by optimizing the storage structure.

[0061] In this embodiment, the data of each business dimension is sorted according to the time dimension (such as date, timestamp) to ensure that the data is arranged in chronological order in the table, thereby optimizing query performance and management efficiency. At the same time, the business data from the same device node is stored in the same subtable. By adding an additional field (device ID) to the data table and using this field for filtering during querying, the database index is ensured to be correctly set to support fast queries based on the time dimension and device node ID.

[0062] See also Figure 3 , is an overall architecture diagram of an energy data preprocessing system in an embodiment of the present application, including:

[0063] The architecture design is divided into two parts: device data consumption and data pre-computation. The data consumption part is used to quickly consume data, and at the same time clean, convert, and classify the data into data that is easy for the system to identify; the data pre-computation part is used to organize the data and summarize it by business.

[0064] The data collected by the data collection program (IOT) is provided to third-party applications for use, while ensuring performance requirements. The principle is to process and convert the data according to business before the third party uses the data, so as to avoid the situation where the data needs to be summarized by business when the third-party application uses the data, which is extremely time-consuming. The data processing process is as follows:

[0065] The data collection system (usually an IOT system) sends the collected data to the message middleware;

[0066] The data consumption service consumes the data stored in the message middleware;

[0067] The consumer service processes the data and stores it in the cache table of the database;

[0068] The data preprocessing service periodically extracts data from the database cache table;

[0069] The data preprocessing service aggregates data into various types of business data according to business aggregation logic;

[0070] The data preprocessing service stores business data in the database for use by other applications.

[0071] Optionally, in some embodiments, before combining the business data according to the time dimension and the business dimension, the method further includes:

[0072] Obtain configuration parameters, and parse the configuration parameters to determine the computing cycle, energy unit, energy type, time dimension, and business dimension of the computing node to be configured;

[0073] The business data is configured according to the calculation cycle, the energy consumption unit, the energy type, the time dimension and the business dimension, and a computing node for processing the business data is generated;

[0074] The cache data is extracted, and the cache data is sent to the computing node for execution to generate summary data.

[0075] It should be noted that by obtaining and parsing the configuration parameters, multiple key attributes of the computing nodes to be configured can be flexibly determined, including computing cycle, energy consumption unit, energy type, time dimension and business dimension, so that the system can adapt to different business scenarios and needs, and functional adjustment and optimization can be achieved without making extensive modifications to the code.

[0076] It should also be noted that, according to the parsed configuration parameters, the business data can be automatically configured and the corresponding computing nodes can be generated. This not only reduces manual intervention, but also improves the degree of automation of data processing. At the same time, since the generation of computing nodes is based on precise configuration parameters, the efficiency and accuracy of data processing can be ensured. By sending cached data to computing nodes for operation, summary data can be generated. In this way, not only data integration is achieved, but also the speed and efficiency of data aggregation are improved through the parallel processing capabilities of computing nodes. By reasonably configuring the computing cycle, time dimension and other parameters of the computing nodes, the utilization of computing resources can be optimized. For example, the number and configuration of computing nodes can be dynamically adjusted according to the actual needs of business data and the complexity of computing tasks, thereby avoiding waste and overuse of resources.

[0077] Through the above methods, by flexibly configuring computing nodes, automated data processing and efficient data aggregation, rapid processing and accurate analysis of business data are achieved, improving the flexibility, efficiency, resource utilization, scalability and maintainability of the system.

[0078] Optionally, in some embodiments, configuring the service data according to the configuration parameters to generate a computing node for processing the service data includes:

[0079] If it is monitored that the time dimension corresponding to the cache data of any of the device nodes is greater than the preset time dimension, the time dimension is split into multiple calculation cycles according to the preset calculation cycle;

[0080] Each of the computing cycles creates a computing node according to the business dimension, and the number of the computing nodes is determined by the product of the number of the computing cycles and the type of the business dimension;

[0081] A computing task is configured for each computing node, and input parameters are distributed to the computing nodes according to the computing task, so that each computing node executes the computing tasks in parallel.

[0082] It should be noted that by monitoring the cache data time dimension of the device node and comparing it with the preset time dimension, it will be split once it exceeds the time dimension. In this way, it can avoid a single task processing data with too long a time span, reduce the complexity and execution time of a single task, and thus improve the overall data processing efficiency.

[0083] The split time dimension is used to create multiple computing nodes according to the preset computing cycle, and the number of computing nodes is further determined according to the business dimension. This design makes full use of the parallelism of computing resources. At the same time, each computing node is responsible for processing a part of the data and executing computing tasks in parallel, which significantly improves the system throughput and can complete the processing of large amounts of data in a shorter time.

[0084] The number of computing nodes is dynamically determined based on the number of computing cycles and the type of business dimension. Through this flexible resource configuration method, the system can use computing resources more efficiently, avoid resource waste, and ensure that each computing node can be fully utilized, thereby improving resource utilization efficiency.

[0085] By executing multiple computing tasks in parallel and distributing input parameters to different computing nodes, each node focuses on processing a part of the data. This division of labor and cooperation helps reduce errors and interference in the computing process, thereby improving the accuracy and reliability of the computing results.

[0086] In this embodiment, by taking into account more possible expansion needs, as the business volume grows and the amount of data increases, the number of computing nodes can be easily increased by increasing the number of computing cycles or expanding the type of business dimensions, thereby meeting higher processing requirements and enhancing the scalability and flexibility of the system.

[0087] Optionally, in some embodiments, configuring a computing task for each computing node includes:

[0088] Call the service interface (i.e., data aggregation interface) to configure the extension point and expand the business logic of the custom business;

[0089] Decomposing the business logic of the custom business into multiple computing subtasks;

[0090] The computing subtask is configured to each computing node (ie, a node generation interface) through a custom class loader, and each computing subtask is loaded to generate business data.

[0091] See Figure 5 , a flow chart of node generation interface in an energy data preprocessing method in an embodiment of the present application, please refer to Figure 6 , is a data aggregation interface flow chart in an energy data preprocessing method in an embodiment of the present application. It should be noted that by calling the service interface to configure the extension point, the business logic of the custom business can be flexibly accessed and expanded. This mechanism enables the system to easily cope with changing business needs without large-scale modifications to the core architecture of the system. Decomposing the custom business logic into multiple computing subtasks helps to achieve parallel processing. Each computing subtask can be executed independently on the computing node, thereby improving the speed and efficiency of data processing. This task decomposition strategy also enables the system to better utilize computing resources, avoiding the problem of a single task occupying too many resources and causing system performance to deteriorate.

[0092] Using custom class loaders to configure computing subtasks to computing nodes allows for fine-grained control of the class loading process. This helps ensure system security and stability and prevents the loading and execution of malicious code; at the same time, custom class loaders also increase flexibility, allowing classes to be dynamically loaded and unloaded based on actual needs, thereby optimizing system resource utilization. With the support of parallel processing and custom class loaders, business data can be generated more quickly; this not only improves the efficiency of data processing, but also shortens the cycle of data generation, allowing business decisions to be made more quickly based on the latest data.

[0093] Through the above methods, by configuring extension points through service interfaces, task decomposition and parallel processing, and the use of custom class loaders, flexible expansion of custom business logic, rapid generation of business data, and maintainability and scalability of the system are achieved, which together improve the overall performance of the system and user experience.

[0094] See also Figure 4 , is a complete flow chart of an energy data preprocessing method in an embodiment of the present application, wherein the converted data is extracted and summarized according to the business dimension, and the data summary process is standardized by defining the overall summary process. At the same time, constraints and guidance are provided for expansion, making business expansion easier. The overall process is as follows:

[0095] A parses the configuration according to the input configuration

[0096] A1, configuration is used to solve the problem that the energy consumption calculation formula has been adjusted in some cases. Since the historical data will be different from the calculation results of the current calculation formula, the historical data needs to be recalculated;

[0097] A2, parses the configuration, which includes calculation cycle, energy consumption unit, energy type, time dimension, and business dimension. The calculation cycle takes the last 15 minutes, all energy consumption units, all energy types, all time dimensions, and all business dimensions.

[0098] A3, after the preprocessing program is triggered, first, the attribution time of the last processed data is obtained as the start time of this calculation, and the end time is taken as the hour and second 15 minutes before the current time. This solves the problem that some data is not calculated due to system downtime for some reasons;

[0099] A4: All parameters will be used as input parameters in subsequent processing nodes.

[0100] B generates business data aggregation processing nodes according to the configuration

[0101] B1, after the program enters the node generation phase, it will generate nodes according to the calculation cycle, time dimension, and business dimension in the configuration;

[0102] B2: If the business dimension parameters are not configured, a computing node is created for each business dimension; if the business dimension parameters are configured, a computing node is created for the specified business dimension;

[0103] B3: Since the table / record will be locked when data is stored and historical data is deleted, in order to reduce the locking of the table without affecting other business data usage, the long calculation cycle is split into several short calculation cycles according to a certain logic. The splitting logic is as follows: the minute dimension is split into 4 hours as a cycle, the hour dimension is split into 12 hours, the day dimension is split into 1 month (dynamic), the month dimension is split into 12 months, and the year dimension is not split;

[0104] B4, each time dimension will create a computing node for the business dimension in B2, and the total number of nodes is the number of time dimension splits * the number of business dimensions;

[0105] B5, each computing node will generate a computing task, and other parameters will be used as input parameters of each computing node for use in business aggregation.

[0106] C Extract cached business data

[0107] D Execute each data aggregation node

[0108] EStorage business summary data

[0109] In this embodiment, the above energy data preprocessing system has the following technical effects:

[0110] 1) High scalability. For different business scenarios, non-invasive customization can be performed according to actual conditions without affecting the original business logic. At the same time, customization is simple. By implementing specified interfaces, it can support multi-scenario and multi-dimensional customization.

[0111] 2) Data consumption is fast. The business logic of consuming the collected device data on the server side is simple and the consumption speed is fast. After the service is interrupted, it can continue to be consumed at the next startup without affecting the continuity of the data.

[0112] 3) High performance: By optimizing the data storage structure and reducing the frequency of use of the formula engine, the performance of various data calculations can be greatly improved, while also improving the query performance of various data.

[0113] Optionally, in some embodiments, after storing the same type of business data in the same data table of the database, the method further includes:

[0114] In response to the service request instruction, access the database to determine associated service data, wherein the associated service data includes a plurality of sub-data;

[0115] Determine the standard deviation of the associated business data and the residual error of each of the sub-data, and determine the random error distribution range based on a preset multiple and the standard deviation;

[0116] Determine whether the residual error is outside the random error distribution range; when the residual error is outside the random error distribution range, remove the sub-data corresponding to the residual error to obtain the target business data corresponding to the business request instruction.

[0117] It should be noted that a business request instruction is received to access the database to obtain specific business data; based on the information in the business request instruction, the database is accessed and the associated business data is retrieved, wherein the associated business data may be a collection or may contain multiple sub-data items, each of which is a part or a record of the entire data set.

[0118] By calculating the standard deviation of the entire associated business data set, a statistic that measures the degree of dispersion of the data set is obtained, which reflects the average difference between each data point in the data set and its mean. For each sub-data item, its residual error is calculated, where the residual error usually refers to the difference between the observed value and the predicted value (or theoretical value), which is understood as the difference between the sub-data item and a certain ideal or expected value.

[0119] A preset multiple (such as 2 times, 3 times, etc.) is used to multiply the standard deviation to obtain a threshold value, which defines the range of random error distribution. That is, if the error is within this range, it is determined to be a normal random fluctuation. If the error exceeds this range, it is determined to be an abnormality or error.

[0120] For the residual error of each sub-data item, determine whether it exceeds the calculated random error distribution range; if the residual error of a sub-data is outside the random error distribution range, the sub-data is determined to be abnormal and is removed from the data set.

[0121] Repeat the above steps until all abnormal sub-data are eliminated. Finally, by cleaning and verifying the target business data, the quality and accuracy of the data are effectively improved. By eliminating erroneous or abnormal data, the reliability and effectiveness of subsequent business processing and analysis are ensured.

[0122] See also Figure 7 , is a schematic diagram of the structure of an energy data preprocessing system in an embodiment of the present application, comprising:

[0123] The acquisition module 701 is used to acquire energy data of each device node and send the energy data to the message middleware;

[0124] The consumption processing module 702 is used to call the message middleware to perform consumption processing on the energy data, determine the consumption data, and store the consumption data as cache data in a cache table of a database;

[0125] A preprocessing module 703 is used to read the cached data and perform preprocessing to obtain extracted business data, wherein the preprocessing includes cleaning, conversion and classification;

[0126] The data storage module 704 is used to combine the business data according to the time dimension and the business dimension, so that the business data of the same type are stored in the same data table of the database.

[0127] It should be noted that the energy data preprocessing method in the present application corresponds one-to-one to the energy data preprocessing system, and the corresponding technical details, i.e., the technical solutions, refer to the above-mentioned energy data preprocessing method, which will not be described in detail here.

[0128] Through the above method, the energy data is consumed by calling the message middleware, and the consumption data is determined and stored in the cache table of the database. In this way, the efficiency and response speed of data processing can be improved; by performing pre-processing operations such as cleaning, conversion and classification on the cache data, not only the data quality and consistency are improved, but also the accuracy and reliability of the data are improved; the business data is combined according to the time dimension and the business dimension, and stored in the same data table of the database. On the one hand, it is conducive to query, analysis and report generation; on the other hand, through the orderly organization and storage of data, not only the efficiency of data access is improved, but also the business scalability is improved; it can also make it easier to understand business trends, identify potential problems, and formulate corresponding solutions.

[0129] In other embodiments, the present disclosure also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0130] Figure 8 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803, such as executing the method in the above embodiment. In the random access memory 803, various programs and data required for system operation are also stored. The central processing unit 801, the read-only memory 802 and the random access memory 803 are connected to each other through a bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0132] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0133] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit 801, various functions defined in the system of the present application are executed.

[0134] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0135] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0136] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0137] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the method, system and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for preprocessing energy data, characterized in that: include: Obtain energy data of each device node and send the energy data to the message middleware; Calling the message middleware to perform consumption processing on the energy data, determining consumption data, and storing the consumption data as cache data in a cache table of a database; Reading the cached data and performing preprocessing to obtain extracted business data, wherein the preprocessing includes cleaning, conversion and classification; The business data are combined according to the time dimension and the business dimension, so that the business data of the same type are stored in the same data table of the database.

2. The method according to claim 1, characterized in that Before combining the business data according to the time dimension and the business dimension, the method further includes: Obtain configuration parameters, and parse the configuration parameters to determine the computing cycle, energy unit, energy type, time dimension, and business dimension of the computing node to be configured; The business data is configured according to the calculation cycle, the energy consumption unit, the energy type, the time dimension and the business dimension, and a computing node for processing the business data is generated; The cache data is extracted, and the cache data is sent to the computing node for execution to generate summary data.

3. The method according to claim 2, characterized in that Configuring the business data according to the configuration parameters to generate a computing node for processing the business data includes: If it is monitored that the time dimension corresponding to the cache data of any of the device nodes is greater than the preset time dimension, the time dimension is split into multiple calculation cycles according to the preset calculation cycle; Each of the computing cycles creates a computing node according to the business dimension, and the number of the computing nodes is determined by the product of the number of the computing cycles and the type of the business dimension; A computing task is configured for each computing node, and input parameters are distributed to the computing nodes according to the computing task, so that each computing node executes the computing tasks in parallel.

4. The method according to claim 3, characterized in that A computing task is configured for each computing node, including: Call the service interface to configure the extension point and expand the business logic of the custom business; Decomposing the business logic of the custom business into multiple computing subtasks; The computing subtask is configured to each computing node through a custom class loader, and each computing subtask is loaded to generate business data.

5. The method according to claim 1, characterized in that Calling the message middleware to perform consumption processing on the energy data and determine the consumption data includes: Extracting the type and processing requirements of the energy data and storing them in a database; Determining whether the energy data is to be consumed according to the processing requirements of the energy data; If it is determined that the energy data is to be consumed, the queue of the preset topic of the message middleware is called for consumption processing according to the type of the energy data, and the generated consumption data is stored in the pending table; for the pending messages in the pending table, consumption processing is performed in real-time consumption processing mode or timed consumption processing mode, and after the consumption processing of the consumption data is completed, the pending messages in the pending table are cleared.

6. The method according to claim 1, characterized in that Combining the business data according to the time dimension and the business dimension so that the business data of the same type are stored in the same data table of the database includes: Using the business summary interface to collect the business data; Divide the business data according to business dimensions so that the business data of each business dimension is stored in a database using a data table; The business data with the same business type are sorted from front to back according to the time dimension, so that the business data from the same device node are stored in the same subtable in the database.

7. The method according to any one of claims 1 to 5, characterized in that: Read the cached data and perform preprocessing to obtain the extracted business data, including: Performing distributed data cleaning on the cache to obtain cleaned data, wherein the data cleaning includes missing values, deduplication, error correction and outlier correction; Performing distributed data conversion on the cleaned data to determine the converted data in a preset format, wherein the data conversion includes at least one of the following: data standardization, data encoding, data aggregation, and data reconstruction; The converted data is subjected to distributed classification processing to obtain extracted business data, wherein the classification processing includes data segmentation and / or business classification.

8. The method according to any one of claims 1 to 5, characterized in that: After the business data of the same type is stored in the same data table of the database, the method further includes: In response to the service request instruction, access the database to determine associated service data, wherein the associated service data includes a plurality of sub-data; Determine the standard deviation of the associated business data and the residual error of each of the sub-data, and determine the random error distribution range based on a preset multiple and the standard deviation; Determine whether the residual error is outside the random error distribution range; when the residual error is outside the random error distribution range, remove the sub-data corresponding to the residual error to obtain the target business data corresponding to the business request instruction.

9. An energy data preprocessing system, characterized in that: include: An acquisition module, used to acquire energy data of each device node and send the energy data to the message middleware; A consumption processing module, used for calling the message middleware to perform consumption processing on the energy data, determining the consumption data, and storing the consumption data as cache data in a cache table of a database; A preprocessing module, used for reading the cached data and performing preprocessing to obtain extracted business data, wherein the preprocessing includes cleaning, conversion and classification; The data storage module is used to combine the business data according to the time dimension and the business dimension so that the business data of the same type are stored in the same data table of the database.

10. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

Images