Power data storage system and method

Abstract

The invention discloses an electric power data storage system and method, and the system comprises a collection module which is used for packaging the obtained power utilization information data collected by each electric power terminal, and transmitting the packaged message data to a communication module; the communication module is used for analyzing the message data, converting the message data into electricity utilization information data according to an analysis result, and sending the electricity utilization information data to the analysis module; the analysis module is used for carrying out anomaly analysis on the electricity utilization information data and carrying out fitting processing on the electricity utilization information data according to an anomaly analysis result to obtain effective electricity utilization data; the storage module is used for classifying the effective power consumption data based on the popularity information and performing hierarchical storage on the effective power consumption data according to a classification result; and the application module is used for sending a data reading initiating request to the storage module according to user requirements so as to obtain the real-time state of the power grid of the target area. According to the system, the data management efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a system and method for storing power data. Background Art

[0002] With the rapid development of the economy and the overall progress of society, the power industry has become increasingly important in modern society, and its digital transformation process is also accelerating. As the core embodiment of the digital development of the power industry, smart grids achieve all-round and real-time monitoring of the operating status of the power system through a large number of power terminal devices, such as smart meters and substation monitoring devices. During operation, these devices will continuously generate massive amounts of electricity consumption information data, and the data size is growing rapidly at a TB-level rate.

[0003] From the perspective of power system operation and management, efficient processing and utilization of these data are crucial.

[0004] However, existing power data storage systems often use a single database to store and manage power data. This lack of targeted optimization makes the database inefficient in processing various query requests.

[0005] It can be seen that how to improve the existing power storage system and improve data management efficiency has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] The present application provides a power data storage system and method to solve the technical problem of how to improve the existing power storage system to achieve the effect of improving data management efficiency.

[0007] In order to solve the above technical problems, an embodiment of the present application provides a power data storage system, including: Acquisition module, communication module, storage module, analysis module and application module, among which, The acquisition module is used to encapsulate the power consumption information data acquired from each power terminal located in the target area, and send the encapsulated message data to the communication module based on the adaptive transmission protocol; The communication module is used to parse the message data, convert the message data into the power usage information data according to the parsing result, and send the power usage information data to the analysis module based on the message queue; The analysis module is used to perform an abnormality analysis on the power consumption information data, and perform fitting processing on the abnormal data in the power consumption information data according to the result of the abnormality analysis to obtain effective power consumption data; The storage module is used to classify the effective power consumption data based on heat information, and store the effective power consumption data in layers according to the classification results, wherein the heat information includes data access frequency, data timeliness and data importance; The application module is used to initiate a data reading request to the storage module according to user needs to obtain the real-time status of the power grid in the target area.

[0008] As one preferred solution, the communication module further includes a communication front-end unit, and the communication front-end unit is used to: Receiving connection requests sent by each of the power terminals; Parsing the connection request, and verifying the terminal logical address obtained by parsing based on a pre-stored legal address list; When the verification is passed, a corresponding TCP link is allocated to the power terminal, and data transmission is performed according to the TCP link.

[0009] As one of the preferred solutions, the communication front-end unit is also used for: Real-time monitoring of the number of message data transmitted by each of the power terminals per unit time; When the amount of the message data exceeds the preset memory limit threshold, the message data exceeding the memory limit threshold is screened as an object to be processed based on a priority rule; Writing the objects to be processed into a format file in sequence based on a preset file format and naming rules, and uploading the written format file to an object storage service; When the amount of the message data is lower than a preset memory limit threshold, a format file is downloaded from the object storage service for re-parsing.

[0010] As one preferred solution, the analysis module further includes a stream computing unit, wherein the stream computing unit is used to: Receiving the serialized electricity usage information data; Comparing the first device identification information in the power usage information data with the second device identification information in the device archive database, and filtering the power usage information data according to the comparison result to obtain first processed data; Logically matching the first processed data with a pre-built task template, and filtering the first processed data according to the matching result to obtain second processed data; The second processed data is subjected to data fitting processing to obtain the effective power consumption data.

[0011] As one preferred solution, the storage module is used for: quantifying the data access frequency, data timeliness and data importance of the electricity consumption information data respectively; According to the quantification results, corresponding weights are assigned to the data access frequency, the data timeliness and the data importance; A comprehensive heat score of the electricity usage information data is calculated according to the quantification result and the weight, the effective electricity usage data is classified according to the comprehensive heat score, and the effective electricity usage data is stored in layers according to the classification result; wherein the classification result includes hot data, cold data and warm data.

[0012] As one preferred solution, the storage module includes: a hot data storage unit, a cold data storage unit and a warm data storage unit, and the storage module is also used for: Real-time monitoring of heat information of the effective electricity consumption data; Based on the heat information, each of the effective power consumption data is evaluated for a degree of coldness or heat, and the effective power consumption data is secondary classified according to the evaluation value according to the evaluation result; According to the secondary classification result, the storage position of the effective power consumption data among the hot data storage unit, the cold data storage unit and the warm data storage unit is dynamically adjusted.

[0013] Another embodiment of the present application provides a method for storing power data, which is applied to the power data storage system as described above, including: Encapsulating the power consumption information data collected by each power terminal located in the target area, and sending the encapsulated message data to the communication module based on the adaptive transmission protocol; Parsing the message data, converting the message data into the power usage information data according to the parsing result, and sending the power usage information data to the analysis module based on the message queue; Performing an abnormality analysis on the power consumption information data, and performing fitting processing on the abnormal data in the power consumption information data according to the result of the abnormality analysis to obtain effective power consumption data; Classifying the effective power consumption data based on heat information, and storing the effective power consumption data in layers according to the classification results, wherein the heat information includes data access frequency, data timeliness and data importance; A data reading request is issued to the storage module according to user needs to obtain the real-time status of the power grid in the target area.

[0014] As one of the preferred solutions, the sending of the encapsulated message data to the communication module based on the adaptive transmission protocol further includes: Real-time monitoring of the number of message data transmitted by each of the power terminals per unit time; When the amount of the message data exceeds the preset memory limit threshold, the message data exceeding the memory limit threshold is screened as an object to be processed based on a priority rule; Writing the objects to be processed into a format file in sequence based on a preset file format and naming rules, and uploading the written format file to an object storage service; When the amount of the message data is lower than a preset memory limit threshold, a format file is downloaded from the object storage service for re-parsing.

[0015] As one of the preferred solutions, the fitting process of the abnormal data in the power consumption information data according to the result of the abnormal analysis includes: in: is the corrected power of the energy meter on day T, The normal power consumption of the electric energy meter in the most recent day from T-1 to T-7. The non-power outage duration of the electricity meter T-1 day, is the rate of change of electric charge; in, The average power consumption of all normal power users on the most recent day of normal power consumption from T-1 to T-7. To make up for the average electricity consumption of all normal electricity users in a day.

[0016] As one of the preferred solutions, the effective power consumption data is classified based on the heat information, and the effective power consumption data is stored in layers according to the classification results, including: quantifying the data access frequency, data timeliness and data importance of the electricity consumption information data respectively; According to the quantification results, corresponding weights are assigned to the data access frequency, the data timeliness and the data importance; A comprehensive heat score of the electricity usage information data is calculated according to the quantification result and the weight, the effective electricity usage data is classified according to the comprehensive heat score, and the effective electricity usage data is stored in layers according to the classification result; wherein the classification result includes hot data, cold data and warm data.

[0017] Compared with the prior art, the beneficial effects of the embodiments of the present application are at least one of the following: (1) This application performs abnormal analysis on electricity consumption information data and can promptly detect abnormal values ​​in the data. These abnormal values ​​may be caused by power terminal failures, communication interference or other factors. If they are not processed, they will affect subsequent data analysis and decision-making. By fitting the abnormal data, the analysis module can correct or supplement the abnormal data according to the historical trends and laws of the data to obtain effective electricity consumption data. This improves the accuracy and reliability of the data and provides more valuable data support for the operation analysis and decision-making of the power grid.

[0018] (2) This application classifies and stores effective electricity consumption data in layers based on heat information. Data is divided into hot data, warm data, and cold data according to data access frequency, timeliness, and importance. This layered storage method can reasonably allocate storage resources according to the different characteristics of data, avoids storing all data on expensive high-speed storage devices, and reduces storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of a power data storage system in one embodiment of the present application; Figure 2 It is a technical architecture diagram of a power data acquisition master station in one embodiment of the present application; Figure 3 It is a communication architecture diagram of a power data acquisition master station in one embodiment of the present application; Figure 4 It is a flowchart of a method for storing power data in one embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0022] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used in this article are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. The term "and / or" used in this article includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0023] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0024] An embodiment of the present application provides a storage system for power data. For details, see Figure 1 , Figure 1 The structure diagram of the power data storage system in one embodiment of the present application is shown, which includes a collection module 11, a communication module 12, an analysis module 13, a storage module 14 and an application module 15, wherein: The acquisition module 11 is used to encapsulate the power consumption information data collected by each power terminal located in the target area, and send the encapsulated message data to the communication module based on the adaptive transmission protocol; Preferably, in one embodiment of the present application, the communication module further includes a communication front-end unit, and the communication front-end unit is used to: Receiving connection requests sent by various power terminals; Parse the connection request and verify the terminal logical address obtained by parsing based on the pre-stored legal address list; When the verification is passed, the corresponding TCP link is allocated to the power terminal, and data is transmitted according to the TCP link.

[0025] The communication front-end unit is at the front end of the power system communication, and will continuously monitor the network port and receive connection requests from various power terminals in the target area. These power terminals can be smart meters, sensors, monitoring equipment, etc. When they start or need to interact with the system, they will send connection requests to the communication front-end unit to establish a communication channel.

[0026] After receiving the connection request, the communication front-end unit will parse the request according to the preset communication protocol. The connection request usually contains relevant information of the terminal, such as the terminal logical address, device type, communication parameters, etc. Through parsing, the communication front-end unit can extract the terminal logical address for subsequent verification operations.

[0027] The communication front-end unit will query the pre-stored legal address list, which records the logical addresses of all power terminals authorized to access the system. The resolved terminal logical address is compared with the legal address list. If the address exists in the list, the verification passes; otherwise, the connection request is considered illegal and the connection is rejected. This verification mechanism can effectively prevent illegal terminals from accessing the system and ensure the security of power system communications.

[0028] When the terminal logical address is verified, the communication front-end unit will allocate a corresponding TCP link for the power terminal. TCP (Transmission Control Protocol) is a connection-oriented, reliable transmission protocol that ensures the accuracy and integrity of data during transmission. The communication front-end unit will establish a TCP connection with the power terminal and transmit power information data through the link. During the data transmission process, the TCP protocol will automatically handle data segmentation, reorganization, confirmation and retransmission operations to ensure that data can be reliably transmitted from the power terminal to the communication front-end unit.

[0029] Preferably, in one embodiment of the present application, the communication front-end unit is further used for: Real-time monitoring of the number of message data transmitted by each power terminal per unit time; When the amount of message data exceeds the preset memory limit threshold, the message data exceeding the memory limit threshold is filtered out based on the priority rule as the object to be processed; Write the objects to be processed into the format file in sequence based on the preset file format and naming rules, and upload the written format file to the object storage service; When the amount of message data is lower than the preset memory limit threshold, the format file is downloaded from the object storage service for re-parsing.

[0030] The communication front-end unit will count the number of message data transmitted by each power terminal in a unit of time (such as per second, per minute) in real time. This can be achieved by setting a counter in the communication front-end unit. Every time a message data is received, the counter of the corresponding power terminal will increase by 1. By monitoring the number of message data in real time, the communication front-end unit can timely understand the data transmission status of each power terminal, providing a basis for subsequent flow control and data processing.

[0031] When the number of message data transmitted by a power terminal in a unit time exceeds the preset memory limit threshold, it means that the data transmission volume of the terminal is too large, which may cause the memory resources of the communication front-end unit to be exhausted, affecting the normal operation of the system. At this time, the communication front-end unit will filter the message data that exceeds the memory limit threshold based on the priority rule. The priority rule can be determined based on factors such as the type, importance, and timeliness of the message data. For example, real-time power grid fault alarm information has a higher priority and will be processed first; while some backup messages of historical data have a lower priority. The filtered message data will be used as the object to be processed.

[0032] The communication module 12 is used to parse the message data, convert the message data into power consumption information data according to the parsing result, and send the power consumption information data to the analysis module based on the message queue; The communication module will first receive the message data sent from the acquisition module based on the adaptive transmission protocol. These message data are obtained by the acquisition module after encapsulating the power consumption information data collected by each power terminal, including power consumption information data and necessary control information and metadata, such as source address, destination address, message type, protocol version, etc.

[0033] Different power terminals may use different communication protocols for data transmission, so the communication module needs to determine the corresponding parsing rules based on information such as the protocol identifier in the message data. Common power communication protocols include Modbus, DL / T645, etc. Each protocol has its own specific data format and encoding rules.

[0034] According to the determined parsing rules, the communication module parses the message data byte by byte or field by field. During the parsing process, the power consumption information data part will be extracted, and the control information such as the header and the tail will be removed. For example, for a message data that complies with the DL / T645 protocol, the communication module will extract the power consumption information data such as voltage, current, and power from the message according to the format specified by the protocol. During the parsing process, data verification is also required, such as using CRC (cyclic redundancy check) code or other verification algorithms to ensure the accuracy and integrity of the parsed data. If the verification fails, the communication module may require the acquisition module to resend the message data.

[0035] The parsed power consumption information data may use a specific encoding format or data type, and the communication module needs to convert it into a unified data format within the system for subsequent processing and analysis. For example, the voltage data collected by some power terminals may be represented in hexadecimal encoding, and the communication module needs to convert it into a decimal voltage value; or convert the data type from a byte array to a floating point number, etc.

[0036] The communication module will select a suitable message queue to realize the asynchronous transmission of power consumption information data. Common message queues include Kafka, RabbitMQ, etc. When selecting a message queue, you need to consider factors such as its performance, reliability, and scalability. For example, Kafka has the characteristics of high throughput and low latency, which is suitable for processing large-scale data streams; RabbitMQ has rich message routing and queue management functions, which is suitable for scenarios with high requirements for message processing. The communication module needs to configure the message queue, including parameters such as the connection address, port number, and queue name.

[0037] The converted power consumption information data is encapsulated into a message format that the message queue can recognize, usually in JSON or XML format. Then, the communication module sends the message to the specified queue of the message queue. During the sending process, the message queue will assign a unique identifier to each message to facilitate subsequent tracking and management. At the same time, the message queue will provide certain reliability guarantees, such as a message confirmation mechanism, to ensure that the power consumption information data can be reliably sent to the message queue.

[0038] The transmission of power consumption information data through the message queue realizes asynchronous transmission and decoupling between the communication module and the analysis module. The communication module is only responsible for sending the power consumption information data to the message queue without waiting for the processing results of the analysis module. The analysis module can asynchronously obtain the power consumption information data from the message queue for processing based on its own processing capabilities. This method improves the concurrent processing capability and reliability of the system. Even if the analysis module fails or the processing speed is slow, it will not affect the data reception and transmission of the communication module. At the same time, the message queue can also buffer and queue the power consumption information data to avoid data loss.

[0039] The analysis module 13 is used to perform abnormal analysis on the power consumption information data, and perform fitting processing on the abnormal data in the power consumption information data according to the result of the abnormal analysis to obtain effective power consumption data; The core task of the analysis module is to perform abnormal analysis on the power consumption information data and perform fitting processing on the abnormal data to obtain effective power consumption data. Abnormal analysis can identify the parts of the data that do not conform to the normal pattern, while fitting processing corrects or supplements these abnormal data, making the data more accurate and reliable, providing strong support for subsequent power system decision-making and management.

[0040] Preferably, in one embodiment of the present application, the analysis module further includes a flow computing unit, wherein the flow computing unit is used to: Receive serialized electricity consumption information data; Comparing the first device identification information in the power usage information data with the second device identification information in the device archive database, and filtering the power usage information data according to the comparison result to obtain first processed data; Logically matching the first processed data with the pre-built task template, and filtering the first processed data according to the matching result to obtain the second processed data; The second processed data is subjected to data fitting processing to obtain effective power consumption data.

[0041] The power consumption information data is generated by the communication module after parsing and conversion. In order to facilitate transmission and storage in the network, these data will be serialized and converted into a specific format, such as JSON, XML or binary format. The stream computing unit will continue to monitor the data channel sent by the communication module and receive the serialized power consumption information data.

[0042] After receiving the serialized data, the stream computing unit needs to deserialize it and restore it to the original data structure and format for subsequent processing and analysis. This step ensures that the stream computing unit can correctly understand and process the received power consumption information data.

[0043] The power usage information data contains the first device identification information, which is information used to uniquely identify each power device, such as the device number, device name, etc. The device archive database stores the relevant information of all legal power devices, including the second device identification information. The flow computing unit compares the first device identification information in the power usage information data with the second device identification information in the device archive database one by one. If the first device identification information of a certain power usage information data can find matching second device identification information in the device archive database, it means that the data comes from a legal power device, and it is filtered out as the first processing data; conversely, if no matching information is found, it is considered that the data may come from an illegal device or there is a data transmission error, and it is filtered out. This screening mechanism can effectively improve the accuracy and security of the data and prevent illegal data from entering the subsequent processing flow.

[0044] Pre-built task templates are a series of rules and conditions based on the business needs and analysis objectives of the power system. These task templates define the logical requirements that different types of power consumption information data should meet, such as power range within a specific time period, voltage fluctuation threshold, etc.

[0045] The stream computing unit logically matches the first processing data with the pre-built task template. For each piece of first processing data, check whether it meets the rules and conditions defined in the task template. If it does, the data is screened out as the second processing data; if it does not, it is considered that the data may be abnormal or does not meet the current analysis requirements and is excluded. In this way, the stream computing unit can further screen out data that meets specific business requirements, reduce the amount of data to be processed later, and improve processing efficiency.

[0046] There may still be some abnormal data in the second processed data, which may be caused by measurement errors, equipment failures, external interference, etc. The stream computing unit needs to identify these abnormal data through certain algorithms and methods, such as methods based on statistical analysis and methods based on machine learning.

[0047] For the identified abnormal data, the stream computing unit will perform data fitting processing. The purpose of data fitting processing is to correct or supplement the abnormal data based on the historical trends and rules of the data to make it more consistent with the normal power consumption pattern. Common data fitting methods include linear interpolation, polynomial fitting, spline interpolation, etc. For example, if the voltage data at a certain moment is abnormal, the stream computing unit can use the linear interpolation method to calculate a reasonable voltage value to replace the abnormal data based on the voltage data of the device at adjacent moments. After data fitting processing, the power consumption information data obtained is the effective power consumption data, which can be used for subsequent power system analysis, decision-making, and management.

[0048] The storage module 14 is used to classify the effective power consumption data based on the heat information, and store the effective power consumption data in layers according to the classification results, wherein the heat information includes data access frequency, data timeliness and data importance; Preferably, in one embodiment of the present application, the storage module is used for: Quantify the data access frequency, data timeliness and data importance of electricity consumption information data respectively; According to the quantitative results, corresponding weights are assigned to data access frequency, data timeliness and data importance; The comprehensive heat score of the electricity consumption information data is calculated based on the quantification results and weights, the effective electricity consumption data is classified based on the comprehensive heat score, and the effective electricity consumption data is stored in layers based on the classification results; the classification results include hot data, cold data and warm data.

[0049] The storage module records the number of times each effective power consumption data item is accessed within a certain time period (such as one day, one week, or one month). In order to facilitate subsequent calculations and classifications, the number of accesses is divided into different levels. For example, the number of accesses less than 10 times is set as low access frequency, 10-50 times as medium access frequency, and more than 50 times as high access frequency, and the corresponding quantitative values ​​are assigned to each, such as low access frequency is quantified as 1, medium access frequency is quantified as 2, and high access frequency is quantified as 3.

[0050] The data timeliness level is determined based on the data update cycle and business needs. For data with extremely high real-time requirements, such as real-time grid voltage and current data, the update cycle is at the second or minute level, and its timeliness is quantified as a high level, such as a value of 3; for data with an update cycle at the hour or day level, such as daily electricity consumption statistics, the timeliness is quantified as a medium level, and a value of 2; for data with a longer update cycle, such as monthly or annual statistics, the timeliness is quantified as a low level, and a value of 1 is assigned.

[0051] The data is evaluated based on its impact on the operation, safety and business decision-making of the power system. Key data related to the safe and stable operation of the power grid, such as bus voltage and main transformer current, are quantified as high-level and assigned a value of 3 if there is a problem with the data, which may lead to power grid failure. Data that has a certain impact on business operations, such as regional power load data and user power consumption trend data, are quantified as medium-level and assigned a value of 2. Data that has a small impact on business, such as historical power consumption consultation records, are quantified as low-level and assigned a value of 1.

[0052] According to the specific business needs and data characteristics of the power system, corresponding weights are assigned to data access frequency, data timeliness and data importance. For example, in scenarios with high real-time monitoring requirements, data timeliness is more important. The weight of data timeliness can be set to 0.5, the weight of data access frequency to 0.3, and the weight of data importance to 0.2; in scenarios that focus on power grid security, the weight of data importance can be appropriately increased, such as setting it to 0.5, the weight of data access frequency to 0.2, and the weight of data timeliness to 0.3. The storage module calculates the comprehensive heat score of each effective power consumption data item by weighted summation based on the quantification results and the assigned weights. According to the comprehensive heat score, the effective power consumption data is divided into three categories: hot data, warm data and cold data. For example, a comprehensive heat score greater than 2.5 is set as hot data, a score between 1.5 and 2.5 is set as warm data, and a score less than 1.5 is set as cold data.

[0053] Preferably, in one embodiment of the present application, the storage module includes: a hot data storage unit, a cold data storage unit and a warm data storage unit, wherein hot data is usually stored in a high-speed storage device due to its high access frequency, strong timeliness and high importance to ensure that the data can be quickly accessed. The access frequency and timeliness of warm data are relatively moderate, and it is stored in a storage device with a relatively balanced performance and cost. Cold data is rarely accessed, but needs to be stored for a long time, and is suitable for storage in a large-capacity, low-cost storage medium.

[0054] The storage module is also used for: Real-time monitoring of heat information of effective electricity consumption data; Based on the heat information, each effective electricity consumption data is evaluated for its degree of coldness and heat, and the effective electricity consumption data is secondary classified according to the evaluation results; According to the secondary classification result, the storage position of the effective power consumption data among the hot data storage unit, the cold data storage unit and the warm data storage unit is dynamically adjusted.

[0055] Specifically, the storage module monitors the popularity of each valid power consumption data in real time, including changes in data access frequency, data timeliness, and data importance. By continuously recording data access, update cycles, and changes in business needs, the dynamic changes in data popularity can be grasped in a timely manner.

[0056] Based on the real-time monitored heat information, the storage module evaluates the degree of heat and coldness of each effective power consumption data. The comprehensive heat score of each data item is recalculated and secondary classification is performed based on the new score. For example, the data originally stored in the warm data storage unit has a significant increase in access frequency due to recent changes in business needs, resulting in an increase in the comprehensive heat score. After secondary classification, it may be classified as hot data.

[0057] According to the secondary classification results, the storage module will dynamically adjust the storage location of effective power consumption data between hot data storage units, cold data storage units, and warm data storage units. For data that was originally stored in a warm data storage unit, but whose temperature is increased to the hot data standard after secondary classification evaluation, it will be migrated from the warm data storage unit to the hot data storage unit; conversely, if the data that was originally stored in a hot data storage unit has its temperature reduced to the standard of warm data or even cold data, it will be migrated to the corresponding storage unit.

[0058] During the data migration process, the storage module will ensure the integrity and consistency of the data. At the same time, in order to reduce the impact of data migration on system performance, strategies such as batch migration and migration during the system off-peak period will be adopted. By dynamically adjusting the storage location, the storage module can reasonably allocate storage resources according to the actual popularity information of the data, improve data access efficiency and the overall performance of the storage system.

[0059] The application module 15 is used to initiate a data reading request to the storage module according to user requirements to obtain the real-time status of the power grid in the target area.

[0060] As an interactive bridge between the power data storage system and users, the core task of the application module is to understand the specific needs of users and convert these needs into data read requests for the storage module, so as to obtain the real-time status information of the target area power grid. By providing such functions, the application module can meet the diverse needs of different users in terms of power grid monitoring, analysis and decision-making.

[0061] The following is a specific embodiment of the present application based on the above storage system. Figure 2 As shown, Figure 2 A technical architecture diagram of a power data acquisition master station provided for this application specifically includes: an application layer, a service layer, a storage and computing layer, and a communication layer.

[0062] The application layer uses VUE technology to build the front-end interface and UI component library, including public basic component library, business component library, customized components, etc. Through flexible layout and efficient rendering components, it provides functions such as draggable reports for thousands of people; combined with Echarts, D3 and other chart plug-ins, it realizes dynamic display of business data, and provides functions such as collection brain, integrated cockpit, and penetration analysis; based on GIS technology, it realizes spatial data visualization, and provides disaster heat map, load heat map, charging heat map and other functions; based on Axios network library, it efficiently processes network requests, and provides request interception, permission verification, unified error handling and other functions.

[0063] The service layer provides configuration management, service discovery, circuit breakers, routing and other service capabilities in distributed scenarios based on Spring Cloud. The design is as follows: 1) Automatic registration and discovery of services are realized through registration centers such as Naocs, making the calls between services more transparent and flexible; 2) Load balancing solutions are built through groups such as Ribbon or Feign to achieve reasonable distribution of requests among multiple service instances; 3) The Hystrix component provides a fuse and degradation mechanism to effectively prevent the collapse of the entire system due to a service failure; 4) Through components such as Config, centralized management and dynamic refresh of configurations are achieved, greatly improving the flexibility of the application; 5) By building, designing, and developing a series of public services such as unified data access services, cache management services, and permission management services, unified management of the application layer is achieved. On this basis, micro-applications for collection success rate, line loss management, distribution transformer management, power outage analysis, and energy monitoring are built.

[0064] The storage and computing layer is responsible for the classified storage of structured and non-structured data, and has the processing capabilities of small and micro computing jobs, batch offline computing tasks, and real-time computing tasks.

[0065] (1) For storage functions, on the one hand, PostgreSQL (hereinafter referred to as PG) was selected to build a relational data storage platform, and the data was divided into historical database and production database, which mainly stored power meter reading data, load data, event data and analysis results of various businesses. The data of the past two years was stored based on the production environment, the data of 2 to 5 years was stored based on the historical environment, and the data of more than 5 years was transferred to the tape drive; on the other hand, CDH was selected to build a big data cluster. Based on Hive and Hbase, the cluster scale of 2 management nodes + 85 computing nodes was designed. The management node was configured with cpu72 cores, memory 512G, disk 2*1T+2*300G, and the computing node was configured with cpu24 cores, memory 256G, and disk 4*10T+2*300G, which can be used to store 10 years of equipment heartbeat, message data, meter reading data, load data, business analysis details data, etc.

[0066] (2) For computing functions, firstly, we choose the computing function of PG itself to build a small and micro computing component, which is responsible for functions with small computing volume, high real-time requirements and frequent computing, such as power outage analysis, power restoration statistics, and power line loss. Secondly, we choose Spark to build an offline computing platform, which is responsible for the analysis and calculation of power collection success rate, collection completeness rate, and disaster power outage impact range, which are extremely computing-intensive and have low real-time requirements. It forms an analysis of the situation of daily equipment power data reporting and collection, assists the power supply station in solving abnormal equipment, and provides a basis for disaster relief and power restoration for large-scale power outages during disasters. Finally, we choose Flink to build a real-time computing platform. First, it is responsible for real-time cleaning and completion of collected data to ensure high data quality. Second, it carries out data fitting for the situation where equipment leaks data and cannot report it again, ensuring 100% completeness of 96 data points for each meter every 15 minutes, and supporting power spot market trading business.

[0067] From a logical design perspective, the communication layer is divided into three parts: communication gateway, communication front-end, and warehousing service. Kafka message queues are used between services for data interaction and logical decoupling. Zookeeper is used as the registration center within the service to implement task scheduling management. The underlying service uses the Netty framework to implement high-concurrency, high-performance instant messaging. The communication front-end uses a distributed containerized deployment method to achieve horizontal agile expansion. The functions of each part are as follows: (1) The communication gateway is mainly responsible for the maintenance of various communication links of the on-site terminals, and performs multiple handshakes with the equipment to ensure access to the device layer; (2) The communication front-end mainly includes FE and BP, among which FE is mainly responsible for the policy distribution of upstream and downstream original messages, and maintains the real-time working condition information of the on-site terminals; BP is mainly responsible for business logic processing, framing / deframing, asset verification, task data / abnormal alarm events and other business data processing; (3) Data warehousing is mainly responsible for the landing of business data, task data (such as power, load, demand, abnormal alarm) and other data. Specifically, Figure 3 As shown, Figure 3 A schematic diagram of the communication architecture of a power data acquisition master station provided in this application.

[0068] Another embodiment of the present application provides a method for storing power data. For details, see Figure 4 , Figure 4 The figure shows a flow chart of a method for storing power data in one embodiment of the present application, which includes steps S1-S5: S1: encapsulating the power consumption information data collected by each power terminal located in the target area, and sending the encapsulated message data to the communication module based on the adaptive transmission protocol; Preferably, in one embodiment of the present application, the encapsulated message data is sent to the communication module based on the adaptive transmission protocol, further comprising: Real-time monitoring of the number of message data transmitted by each power terminal per unit time; When the amount of message data exceeds the preset memory limit threshold, the message data exceeding the memory limit threshold is screened as an object to be processed based on a priority rule; Writing the objects to be processed into a format file in sequence based on a preset file format and naming rules, and uploading the written format file to an object storage service; When the amount of message data is lower than the preset memory limit threshold, the format file is downloaded from the object storage service for re-parsing.

[0069] S2: Parse the message data, convert the message data into power consumption information data according to the parsing result, and send the power consumption information data to the analysis module based on the message queue; S3: Perform an abnormal analysis on the power consumption information data, and perform fitting processing on the abnormal data in the power consumption information data according to the results of the abnormal analysis to obtain effective power consumption data; Construct the electricity data fitting, and generate fitting values ​​for the missing data through simulation and correction to minimize the impact on productive business. The most critical electricity data of the master station can be calculated through the data reported on day T and the data reported on day T-1. The data items include the total forward active power and the total reverse active power. When data is missing, the data is supplemented and corrected according to the following rules. Assuming that the daily frozen power of the electricity meter on day T cannot be calculated, preferably, in one embodiment of the present application, the abnormal data in the electricity information data is fitted according to the results of the abnormal analysis, including: in: is the corrected power of the energy meter on day T, The normal power consumption of the electric energy meter in the most recent day from T-1 to T-7. The non-power outage duration of the electricity meter T-1 day, is the rate of change of electric charge; in, The average power consumption of all normal power users on the most recent day of normal power consumption from T-1 to T-7. To make up for the average electricity consumption of all normal electricity users in a day.

[0070] Supplementary rules: (1) Normal electricity consumption refers to the user's daily electricity consumption without replenishment and correction, and the user's status on that day is normal operation.

[0071] (2) If the user has not had normal electricity consumption in the past seven days (excluding newly installed meters), the corrected electricity consumption of the most recent day will be used for calculation.

[0072] S4: Classify the effective electricity consumption data based on heat information, and store the effective electricity consumption data in layers according to the classification results, wherein the heat information includes data access frequency, data timeliness, and data importance; Preferably, in one embodiment of the present application, the effective power consumption data is classified based on the heat information, and the effective power consumption data is stored in layers according to the classification results, including: Quantify the data access frequency, data timeliness and data importance of electricity consumption information data respectively; According to the quantitative results, corresponding weights are assigned to data access frequency, data timeliness and data importance; The comprehensive heat score of the electricity consumption information data is calculated based on the quantification results and weights, the effective electricity consumption data is classified based on the comprehensive heat score, and the effective electricity consumption data is stored in layers based on the classification results; the classification results include hot data, cold data and warm data.

[0073] S5: Initiate a data read request to the storage module according to user needs to obtain the real-time status of the power grid in the target area.

[0074] Compared with the prior art, the embodiments of the present application have the following advantages: (1) This application performs abnormal analysis on electricity consumption information data and can promptly detect abnormal values ​​in the data. These abnormal values ​​may be caused by power terminal failures, communication interference or other factors. If they are not processed, they will affect subsequent data analysis and decision-making. By fitting the abnormal data, the analysis module can correct or supplement the abnormal data according to the historical trends and laws of the data to obtain effective electricity consumption data. This improves the accuracy and reliability of the data and provides more valuable data support for the operation analysis and decision-making of the power grid.

[0075] (2) This application classifies and stores effective electricity consumption data in layers based on heat information. Data is divided into hot data, warm data, and cold data according to data access frequency, timeliness, and importance. This layered storage method can reasonably allocate storage resources according to the different characteristics of data, avoids storing all data on expensive high-speed storage devices, and reduces storage costs.

[0076] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A power data storage system, characterized in that: It includes acquisition module, communication module, storage module, analysis module and application module, among which: The acquisition module is used to encapsulate the power consumption information data acquired from each power terminal located in the target area, and send the encapsulated message data to the communication module based on the adaptive transmission protocol; The communication module is used to parse the message data, convert the message data into the power usage information data according to the parsing result, and send the power usage information data to the analysis module based on the message queue; The analysis module is used to perform an abnormality analysis on the power consumption information data, and perform fitting processing on the abnormal data in the power consumption information data according to the result of the abnormality analysis to obtain effective power consumption data; The storage module is used to classify the effective power consumption data based on heat information, and store the effective power consumption data in layers according to the classification results, wherein the heat information includes data access frequency, data timeliness and data importance; The application module is used to initiate a data reading request to the storage module according to user needs to obtain the real-time status of the power grid in the target area.

2. The power data storage system according to claim 1, characterized in that: The communication module further includes a communication pre-unit, and the communication pre-unit is used for: Receiving connection requests sent by each of the power terminals; Parsing the connection request, and verifying the terminal logical address obtained by parsing based on a pre-stored legal address list; When the verification is passed, a corresponding TCP link is allocated to the power terminal, and data transmission is performed according to the TCP link.

3. The power data storage system according to claim 2, characterized in that: The communication front-end unit is also used for: Real-time monitoring of the number of message data transmitted by each of the power terminals per unit time; When the amount of the message data exceeds the preset memory limit threshold, the message data exceeding the memory limit threshold is screened as an object to be processed based on a priority rule; Writing the objects to be processed into a format file in sequence based on a preset file format and naming rules, and uploading the written format file to an object storage service; When the amount of the message data is lower than a preset memory limit threshold, a format file is downloaded from the object storage service for re-parsing.

4. The power data storage system according to claim 1, characterized in that: The analysis module further includes a flow computing unit, wherein the flow computing unit is configured to: Receiving the serialized electricity usage information data; Comparing the first device identification information in the power usage information data with the second device identification information in the device archive database, and filtering the power usage information data according to the comparison result to obtain first processed data; Logically matching the first processed data with a pre-built task template, and filtering the first processed data according to the matching result to obtain second processed data; The second processed data is subjected to data fitting processing to obtain the effective power consumption data.

5. The power data storage system according to claim 1, characterized in that: The storage module is used for: quantifying the data access frequency, data timeliness and data importance of the electricity consumption information data respectively; According to the quantification results, corresponding weights are assigned to the data access frequency, the data timeliness and the data importance; Calculating a comprehensive heat score of the power usage information data according to the quantification result and the weight, classifying the effective power usage data according to the comprehensive heat score, and storing the effective power usage data in layers according to the classification result; The classification results include hot data, cold data and warm data.

6. The power data storage system according to claim 5, characterized in that: The storage module comprises: a hot data storage unit, a cold data storage unit and a warm data storage unit, and the storage module is further used for: Real-time monitoring of heat information of the effective electricity consumption data; Based on the heat information, each of the effective power consumption data is evaluated for a degree of coldness or heat, and the effective power consumption data is secondary classified according to the evaluation value according to the evaluation result; According to the secondary classification result, the storage position of the effective power consumption data among the hot data storage unit, the cold data storage unit and the warm data storage unit is dynamically adjusted.

7. A method for storing power data, applied to the power data storage system according to any one of claims 1 to 6, characterized in that: include: Encapsulating the power consumption information data collected by each power terminal located in the target area, and sending the encapsulated message data to the communication module based on the adaptive transmission protocol; Parsing the message data, converting the message data into the power usage information data according to the parsing result, and sending the power usage information data to the analysis module based on the message queue; Performing an abnormality analysis on the power consumption information data, and performing fitting processing on the abnormal data in the power consumption information data according to the result of the abnormality analysis to obtain effective power consumption data; Classifying the effective power consumption data based on heat information, and storing the effective power consumption data in layers according to the classification results, wherein the heat information includes data access frequency, data timeliness and data importance; A data reading request is issued to the storage module according to user needs to obtain the real-time status of the power grid in the target area.

8. The method for storing power data according to claim 7, characterized in that: The method further comprises sending the encapsulated message data to the communication module based on the adaptive transmission protocol, and further comprising: Real-time monitoring of the number of message data transmitted by each of the power terminals per unit time; When the amount of the message data exceeds the preset memory limit threshold, the message data exceeding the memory limit threshold is screened as an object to be processed based on a priority rule; Writing the objects to be processed into a format file in sequence based on a preset file format and naming rules, and uploading the written format file to an object storage service; When the amount of the message data is lower than a preset memory limit threshold, a format file is downloaded from the object storage service for re-parsing.

9. The method for storing power data according to claim 7, characterized in that: The fitting process of the abnormal data in the power consumption information data according to the result of the abnormal analysis includes: in: is the corrected power of the energy meter on day T, The normal power consumption of the electric energy meter in the most recent day from T-1 to T-7. The non-power outage duration of the electricity meter T-1 day, is the rate of change of electric charge; in, The average power consumption of all normal power users on the most recent normal power consumption day from T-1 to T-7. To make up for the average electricity consumption of all normal electricity users in a day.

10. The method for storing power data according to claim 7, characterized in that: The classifying the effective power consumption data based on the heat information and storing the effective power consumption data in layers according to the classification results includes: quantifying the data access frequency, data timeliness and data importance of the electricity consumption information data respectively; According to the quantification results, corresponding weights are assigned to the data access frequency, the data timeliness and the data importance; A comprehensive heat score of the electricity usage information data is calculated according to the quantification result and the weight, the effective electricity usage data is classified according to the comprehensive heat score, and the effective electricity usage data is stored in layers according to the classification result; wherein the classification result includes hot data, cold data and warm data.

Images