Multi-energy data processing system and processing method
By designing multiple energy data processing systems, using MQ processing units, Redis cache processing units and multi-threaded processing units to decouple data processing, the database storage pressure and system stability problems in the prior art are solved, and the energy data is efficient Processing and storage ensures the scalability and stability of the system.
Patent Information
- Application Number
- CN202410986212.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, with the increase of monitoring equipment in the building, the storage of relational databases is getting bigger and bigger, and the system maintenance and upgrade is becoming more and more difficult, resulting in increased read and write pressure on servers, and changes, abnormalities or crashes of relational databases are being made to the entire system. Have a destructive effect.
A variety of energy data processing systems have been designed, including data acquisition module, data processing module and data storage module. The MQ processing unit, Redis cache processing unit and multi-threaded processing unit are used to decouple data through publishing subscription mode and cache mechanism. Process, reduce database access frequency, and use upward compensation algorithm to ensure data continuity and real-time.
It realizes efficient processing and storage of energy data, supports system scalability and stability, reduces the read and write pressure of the database server, ensures real-time and continuity of data, and avoids the risk of system crash.
Smart Images

Figure CN118939713A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of energy consumption data processing, and in particular to a system and method for processing multiple energy data. Background Art
[0002] With the rapid development of society, the operating costs of enterprises have increased due to excessive energy consumption, and most enterprises do not monitor energy consumption, so the energy consumption problem has become a difficult problem for tens of millions of enterprises. Therefore, in order to help enterprises achieve intelligent energy saving and help enterprises to control energy consumption costs in real time, an energy consumption monitoring system has been launched. Through sensor equipment, energy operation, failure, consumption and other data are obtained in real time at a high frequency, and the data is analyzed and mined in depth to achieve accurate and intuitive energy control. The law of energy consumption is used to help enterprises manage energy conservation and achieve systematic energy conservation and consumption reduction.
[0003] As time goes by, the number of monitoring devices in buildings increases, the storage of relational databases becomes larger and larger, and the system becomes more and more difficult to maintain and upgrade. Each application accesses the database to obtain data, which greatly increases the read and write pressure of the server. If the relational database changes, becomes abnormal, or crashes, it will have a destructive impact on the entire system.
[0004] As mentioned above, we have designed a multi-energy data processing system and processing method to solve the above problems. Summary of the invention
[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a multi-energy data processing system and processing method.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] A multi-energy data processing system includes a system body connected to a cloud server, wherein the system body includes a data acquisition module, a data processing module, an energy consumption calculation module and a data storage module;
[0008] The data acquisition module includes a sensor and a gateway. The gateway and the sensor are connected via data. After the sensor collects data, the data is collected through the gateway.
[0009] The data processing module includes an MQ processing unit, a Redis cache processing unit, a multi-thread processing unit, and a warehousing processing unit;
[0010] The MQ processing unit is connected to the sensor by signal, the Redis cache processing unit and the MQ processing unit are both connected to the multi-thread processing unit by signal, and the multi-thread processing unit is connected to the warehousing processing unit by signal;
[0011] The energy consumption calculation module includes calculating and storing energy consumption data for the year, month, week, day and hour;
[0012] The data storage module includes a database for storing calculated data.
[0013] A processing method for a multi-energy data processing system includes the following steps:
[0014] S1. Initialization point data: When the system starts, the corresponding time of the initialization point, including year, month, week, day, and hour, is read from the database for the first point data within the time period whose value is greater than 0 and inserted into the Redis cache;
[0015] S2, subscribe to real-time data: through the publish-subscribe mode of MQ middleware, the system subscribes to the real-time sensor data collected by the gateway;
[0016] S3, cache data update: After receiving the real-time data in MQ, determine whether the current data is greater than 0 and the cached data is equal to 0. If so, update the data in the Redis cache;
[0017] S4, data processing and storage: start parallel thread processing and store data into the database, start energy consumption calculation thread, and calculate energy consumption data for year, month, week, day, and hour respectively;
[0018] S5, energy consumption calculation: start the energy consumption calculation thread, and calculate the energy consumption data of year, month, week, day and hour according to the customized energy consumption calculation method;
[0019] S6. Energy consumption statistics: Configure the statistical time interval, regularly calculate and store energy consumption data, and store the calculation results in the database.
[0020] Preferably, the system body also includes an energy consumption display module, which includes a display connected to the cloud server, and the display is used to display real-time consumption trend graphs, regional data consumption graphs, and basic data for drill-down analysis results.
[0021] Preferably, in step S1, the corresponding time year, month, week, day and hour of the initialization point when the system is started include:
[0022] Year = first day of the current year, month = first day of the current month, week = first day of the current week, day = current day, hour = current hour.
[0023] Preferably, in step S2, when MQ publishes and subscribes to the service, the data is decoupled, asynchronous thread processing is performed after receiving, and data is stored in parallel and written to the cache; during the storage process, data is stored once every 5 minutes, historical data is stored, the basic value of the data can be tracked, and the basic data of the trend chart is provided to be displayed on the display.
[0024] Preferably, in step S3, the first initial data is cached, and the periodic time changes to replace the last cached initial data; the real-time value is compared with the cached initial value and replaced in sequence.
[0025] Preferably, in step S3, when network fluctuations cause MQ subscription failure, the system uses an upward compensation algorithm:
[0026] Get the latest data from the Redis cache as the current latest data to ensure data continuity and real-time performance;
[0027] If the data in the Redis cache is older, set a tolerance to allow the old data to be used for calculation within a certain period of time until MQ reconnects successfully.
[0028] Preferably, in step S6, the energy consumption data within the time period = the current real-time value - the cache value.
[0029] Compared with the prior art, the invention has the following beneficial effects: it can reuse technology for all smart devices, providing strong data support for energy management of enterprises; it can decouple data from each other, supporting unlimited expansion between applications. It provides basic data storage, which can support the use of various legends on the display end. The storage mode is divided into cache and database, supporting the calculation method and data analysis of hybrid data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 A schematic diagram of data flow for basic data processing into cache and storage in the present invention;
[0031] Figure 2 It is a flow chart of the energy consumption data calculation process in the basic data calculation in the present invention. DETAILED DESCRIPTION
[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0033] Reference Figure 1 , a multi-energy data processing system and processing method, including a system body connected to a cloud server, the system body including a data acquisition module, a data processing module, an energy consumption calculation module and a data storage module;
[0034] The data acquisition module includes a sensor and a gateway. The gateway and the sensor are connected by data. After the sensor collects data, the data is collected through the gateway; the data processing module includes an MQ processing unit, a Redis cache processing unit, a multi-threaded processing unit, and a warehousing processing unit; the MQ processing unit is signal-connected to the sensor, the Redis cache processing unit and the MQ processing unit are both signal-connected to the multi-threaded processing unit, and the multi-threaded processing unit is signal-connected to the warehousing processing unit; the energy consumption calculation module includes calculating and storing energy consumption data of year, month, week, day, and hour; the data storage module includes a database for storing the calculated data.
[0035] Among them, the system body also includes an energy consumption display module, the energy display module includes a display connected to the cloud server, and the display is used to display real-time consumption trend charts, regional data consumption charts and basic data for drill-down analysis results.
[0036] A processing method for a multi-energy data processing system includes the following steps:
[0037] S1. Initialization point data: When the system starts, the corresponding time of the initialization point, the time includes: (year = the first day of the year (such as: 2022-01-01 00:00:00), month = the first day of the month (2022-11-01 00:00:00), week = the first day of this week (2022-11-14 00:00:00), day = the current day (2022-11-17 00:00:00), hour = current hour (2022-11-17 01:00:00), read the first point data within the time period and the value greater than 0 from the database, and insert it into the Redis cache;
[0038] S2, subscribe to real-time data: through the publish-subscribe mode of MQ middleware, the system subscribes to the real-time sensor data collected by the gateway;
[0039] S3, cache data update: After receiving the real-time data in MQ, determine whether the current data is greater than 0 and the cached data is equal to 0. If so, update the data in the Redis cache;
[0040] S4, data processing and storage: start parallel thread processing and store data into the database, start energy consumption calculation thread, and calculate energy consumption data for year, month, week, day, and hour respectively;
[0041] S5, energy consumption calculation: start the energy consumption calculation thread, calculate the energy consumption data of the year, month, week, day and hour according to the customized energy consumption calculation method, and perform real-time data consumption results according to the set time, and verify / repair the last calculation results;
[0042] S6. Energy consumption statistics: configure the statistical time interval, regularly calculate and store energy consumption data, and store the calculation results in the database / time series library, where the energy consumption data within the time period = current real-time value - cache initial value of the corresponding time period.
[0043] It should be noted that in step S2, when MQ publishes and subscribes to services, the data is decoupled, asynchronous thread processing is performed after receiving, and data is stored in parallel and written to the cache; during the storage process, data is stored once every 5 minutes, historical data is stored, the basic data values can be tracked, and trend chart basic data is provided to be displayed on the display. The cache middleware is used to reduce the frequency of database access and improve system performance.
[0044] Among them, in the step S3, the first initial data is cached, and the cycle time changes, replacing the last cached initial data; the real-time value is compared with the cached initial value and replaced in sequence; when the network fluctuation causes the MQ subscription to fail, the system uses an upward compensation algorithm:
[0045] Get the latest data from the Redis cache as the current latest data to ensure data continuity and real-time performance;
[0046] If the data in the Redis cache is older, set a tolerance to allow the old data to be used for calculation within a certain period of time until MQ reconnects successfully.
[0047] Through the above technical content, the calculation is repaired every 5 minutes to solve the problems of data jumping and abnormal data reading. Real-time value and cache point processing can be carried out without interrupting the calculation. The initial value is calculated during acquisition, and the acquisition timeout will not be caused by excessive data volume. The cache value is the latest data and does not affect the calculation of new points. It can also calculate the energy consumption data with the minimum error in the corresponding time period.
[0048] Embodiment 1
[0049] Technical Solution 1 of this embodiment: In the implementation, in view of the continuous increase of devices and the possibility of server downtime, an MQ queue subscription is added to timely process the collected real-time sensor data, and microservice distributed deployment is adopted to disperse system risks and handle unexpected disasters in the event of a server outage.
[0050] Technical solution 2 of this embodiment: To ensure the real-time performance of data, the subscription mode of MQ queue is adopted. After MQ receives the data, it triggers the bound subscription event, updates the data to the cache of basic data, prepares for the subsequent storage of basic data, and compares the acquired data with the initial cache data in parallel. If the cached data is greater than the current real-time value, the comparison data cache is updated. If the current value is greater than the cached value, the cache is updated; the latest data is provided for the comparison cache for subsequent calculations.
[0051] The MQ queue stores the real-time data read by the sensor, and uses the MQ first-in-first-out mechanism to implement the first-in-first-out data structure of the basic data. Through asynchronous processing, the order of data time is solved, the data is decoupled, and the direct multi-level correlation between data is reduced. The flow peak can be controlled, and the throughput of the energy consumption calculation system can be controlled to remain stable during peak hours. This ensures the safe and stable operation of the system.
[0052] Technical solution three of this embodiment: in the relational database, the database is too large, the program batch obtains relational data timeout, the basic continuity data is missing due to connection anomalies, the calculation results are inconsistent with the actual results, and the calculation results are not timely.
[0053] This embodiment uses MQ queue automatic subscription triggering, uses Redis caching for data with excessive consumption, and uses the previous data filling method to ensure data continuity through cache storage value. It decouples programs and realizes the method of calculating annual energy consumption, monthly energy consumption, weekly energy consumption, daily energy consumption, and hourly energy consumption by taking values from the database, server, and energy consumption monitoring system.
[0054] Among them, year = the first day of the year (such as: 2022-01-01 00:00:00), month = the first day of the month (2022-11-01 00:00:00), week = the first day of the week (2022-11-14 00:00:00), day = the current day (2022-11-17 00:00:00), hour = the current hour (2022-11-17 01:00:00).
[0055] Technical solution 4 of this embodiment: In the case of unexpected server downtime or too many equipment points causing high concurrency, the cache server is used to reduce the database access concurrency pressure and maintain the stability of the database server. The first basic data of all equipment points that is not 0 is cached as hot standby. When the acquired point data is always 0, the data cache is 0. Caching 0 for non-existent points is to prevent cache penetration. Impacting the server causes delays and anomalies in the system calculation logic.
[0056] Technical Solution 5 of this embodiment: Configure the calculation (interval) repair data time (default 5 minutes). Multithreaded processing operation starts multithreading, and the cache update thread is set to the highest level. All data comes out of MQ, first passes the cache verification, and then the basic database storage and calculation operations are executed in parallel.
[0057] Thread 1: Read the real-time data in the cache and store it directly in the database. This provides a complete and accurate data foundation for display, deep drilling, dimensional analysis, real-time statistics, and real-time query and analysis.
[0058] Thread 2: Compare the current MQ real-time value with the stored cache value. If the real-time value is greater than the initial value of the cache value and the new upper point is greater than 0, it is added to the cache. Otherwise, the current thread ends.
[0059] Thread 3: Calculate annual energy consumption, monthly energy consumption, weekly energy consumption, daily energy consumption, and hourly energy consumption. Consumption value = value in current real-time cache - first initial value in cache.
[0060] In this embodiment, combined with Figure 1 , a data storage method of intelligent operation digital management system that stores the collected data in a hybrid data storage (using cache, memory, and database storage) to improve the performance of the program. The data processing module includes MQ processing unit, Redis cache processing unit, multi-thread processing unit, and storage processing unit. Through the microservice architecture, an efficient and reasonable storage method for collected data is achieved to reduce the coupling between data; Figure 2 Energy consumption calculation provides a data basis. At the same time, it reduces the frequency of database usage in the program and greatly reduces the read consumption of the database server.
[0061] What needs to be specifically explained is that when processing in MQ, the publish-subscribe model is adopted, which loosely couples the publisher and the subscriber, and also decouples them in time. The publisher notifies the subscriber in real time, and supports the subscription system to receive the data of the publisher (collected by the gateway) in real time; it greatly improves the system response speed, greatly improves the speed of the entire data entering the warehouse and cache, and ensures the real-time nature of the data.
[0062] It should be specifically noted that the above cache processing improves the system's data acquisition performance, and uses an upward filling method to supplement data lost in collection or MQ data loss in the event of data collection interruption, thereby ensuring the continuity of basic data.
[0063] More specifically, in the multi-threaded processing mode, in order to improve the CPU performance and maximize the CPU performance, a multi-threaded parallel processing mode is adopted. In the calculation, the energy consumption of the year, month, week, day and hour is processed in parallel; the computer runs efficiently, and there is no waiting or delay between calculations.
[0064] The basic principles, main features and advantages of the present invention are shown and described above. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.
Claims
1. A multi-energy data processing system, comprising a system body connected to a cloud server, characterized in that: The system body includes a data acquisition module, a data processing module, an energy consumption calculation module and a data storage module; The data acquisition module includes a sensor and a gateway. The gateway and the sensor are connected via data. After the sensor collects data, the data is collected through the gateway. The data processing module includes an MQ processing unit, a Redis cache processing unit, a multi-thread processing unit, and a warehousing processing unit; The MQ processing unit is connected to the sensor by signal, the Redis cache processing unit and the MQ processing unit are both connected to the multi-thread processing unit by signal, and the multi-thread processing unit is connected to the warehousing processing unit by signal; The energy consumption calculation module includes calculating and storing energy consumption data for the year, month, week, day and hour; The data storage module includes a database for storing calculated data.
2. A processing method based on the energy consumption data processing system according to claim 1, characterized in that: The following steps are included: S1. Initialization point data: When the system starts, the corresponding time of the initialization point, including year, month, week, day, and hour, is read from the database for the first point data within the time period whose value is greater than 0 and inserted into the Redis cache; S2, subscribe to real-time data: through the publish-subscribe mode of MQ middleware, the system subscribes to the real-time sensor data collected by the gateway; S3, cache data update: After receiving the real-time data in MQ, determine whether the current data is greater than 0 and the cached data is equal to 0. If so, update the data in the Redis cache; S4, data processing and storage: start parallel thread processing and store data into the database, start energy consumption calculation thread, and calculate energy consumption data for year, month, week, day, and hour respectively; S5, energy consumption calculation: start the energy consumption calculation thread, and calculate the energy consumption data of year, month, week, day and hour according to the customized energy consumption calculation method; S6. Energy consumption statistics: Configure the statistical time interval, regularly calculate and store energy consumption data, and store the calculation results in the database.
3. A multi-energy data processing system according to claim 1, characterized in that: The system body also includes an energy consumption display module, which includes a display connected to the cloud server, and the display is used to display real-time consumption trend graphs, regional data consumption graphs, and basic data for drill-down analysis results.
4. The processing method of a multi-energy data processing system according to claim 2, characterized in that: In step S1, the system startup will initialize the corresponding time year, month, week, day, and hour of the point including: Year = first day of the current year, month = first day of the current month, week = first day of the current week, day = current day, hour = current hour.
5. The processing method of a multi-energy data processing system according to claim 2, characterized in that: In step S2, when MQ publishes and subscribes to services, the data is decoupled, asynchronous thread processing is performed after receiving, and data is stored in parallel and written to the cache; during the storage process, data is stored once every 5 minutes, historical data is stored, the basic values of the data can be tracked, and the basic data of the trend chart is provided to be displayed on the display.
6. The processing method of a multi-energy data processing system according to claim 2, characterized in that: In step S3, the first initial data is cached, and the periodic time changes to replace the last cached initial data; the real-time value is compared with the cached initial value and replaced in sequence.
7. The processing method of a multi-energy data processing system according to claim 2, characterized in that: In step S3, when network fluctuations cause MQ subscription failure, the system uses an upward compensation algorithm: Get the latest data from the Redis cache as the current latest data to ensure data continuity and real-time performance; If the data in the Redis cache is older, set a tolerance to allow the old data to be used for calculation within a certain period of time until MQ reconnects successfully.
8. The processing method of a multi-energy data processing system according to claim 2, characterized in that: In step S6, the energy consumption data within the time period = the current real-time value - the cache value.
