Mass power grid data collecting and processing method and system and storage medium

Abstract

The invention provides a mass power grid data collecting and processing method and system and a storage medium, and the method comprises the steps: pre-configuring HES labels, FTP accounts, FTP folders and analysis rules corresponding to different HES systems, and creating a mapping relation; different HES systems upload the power grid data to corresponding FTP folders through FTP accounts of the HES systems; when monitoring that a new file is added in the FTP folder, generating a notification message and sending the notification message to the Netty server; the notification message is stored in a local KAFKA blocking queue; reading newly-added file content from the FTP folder through a Worker Group thread, converting the newly-added file content into a file stream, and sending the file stream to a content analysis channel; and the content analysis channel performs analysis processing on the original power grid data in the file stream according to a corresponding analysis rule, and converts the original power grid data into standardized power grid data. Massive power grid data of multiple docking parties and multiple docking formats can be efficiently and reliably processed.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for collecting and processing massive power grid data, a system thereof, and a storage medium. Background Art

[0002] Grid data acquisition is a key component of power system operation and management. It involves collecting real-time data from various sensors and smart devices (such as meters, relays, circuit breakers, etc.) on the grid. This data is essential for monitoring, controlling, optimizing and safe operation of the grid. On this basis, some advanced measurement architectures have been derived to provide more detailed power consumption data and support advanced applications such as demand response, load management and power quality management. However, the use of traditional technical data processing technology implementation solutions can only achieve basic monitoring of grid data, which is difficult to meet the grid system's needs for data intelligence and the establishment of advanced measurement systems. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, one object of the present invention is to provide a method for collecting and processing massive power grid data, which can efficiently and reliably process massive power grid data of multiple docking parties and multiple docking formats.

[0004] The second object of the present invention is to provide a massive power grid data acquisition and processing system, which has the advantages of high throughput, scalability, high reliability, and standard storage.

[0005] The third object of the present invention is to provide a computer-readable storage medium that can realize efficient and reliable processing of massive power grid data in multiple docking parties and multiple docking formats.

[0006] To achieve the above-mentioned purpose, a first embodiment of the present invention proposes a method for collecting and processing massive power grid data, comprising: Pre-configure HES tags, FTP accounts, FTP folders and parsing rules corresponding to different HES systems; Create mapping relationships between different HES systems and corresponding HES tags, FTP accounts, FTP folders, and parsing rules; Different HES systems upload their power grid data to the corresponding FTP folders through their FTP accounts; When a new file is detected in an FTP folder, a notification message with the corresponding HES tag is generated and sent to the Netty server. The Netty server stores the notification message in the local KAFKA blocking queue; The KAFKA blocking queue forwards the notification message to the content parsing channel, and reads the newly added file content from the FTP folder through the Worker Group thread, and converts it into a file stream and sends it to the content parsing channel; The content parsing channel parses the original power grid data in the file stream according to the mapping relationship and the parsing rule corresponding to the HES tag carried in the notification message, and converts it into standardized power grid data; Store standardized power grid data in a distributed database.

[0007] According to a method for collecting and processing massive power grid data in an embodiment of the present invention, for different docking parties and different docking formats, by configuring corresponding HES tags, FTP accounts, FTP folders and parsing rules, and creating corresponding mapping relationships, accurate and reliable distinction and targeted processing can be achieved; by monitoring new files through the Netty server, and using the local KAFKA blocking queue and content parsing channel, based on the distributed process concurrent processing mechanism and mapping relationships, the new data is accurately, timely and reliably pulled and targeted processing is performed, which can achieve targeted differential preprocessing of massive power grid data of different access parties and different docking formats, and ensure the subsequent availability and effectiveness of the data at the storage layer and application layer.

[0008] In addition, the method for collecting and processing massive power grid data proposed in the above embodiment of the present invention may also have the following additional technical features: Optionally, the changes of the FTP folder are monitored through a Shell script.

[0009] Optionally, the Netty server stores the notification message into a local KAFKA blocking queue in a thread-free manner.

[0010] Optionally, the reading of the newly added file content from the FTP folder through the Worker Group thread and converting the file stream into a file stream and sending it to the content parsing channel includes: The singleton consumer thread in the KAFKA blocking queue reads the notification message and calls the load balancing module to forward the notification message to a Netty client; The Netty client converts the notification message into a file read request corresponding to each newly added file and distributes it to the Worker Group thread; The Worker Group thread that receives the file read request reads the corresponding newly added file content in the FTP folder, converts it into a file stream and sends it to the content parsing channel.

[0011] Optionally, the parsing process includes data cleaning and formatting.

[0012] Optionally, storing the standardized power grid data in a distributed database includes: Sending the standardized power grid data to the KAFKA exclusive queue; The KAFKA exclusive queue performs streaming calculations on the standardized power grid data through the big data SPARK task, and stores the calculation results in a distributed database.

[0013] Optionally, it also includes: Access the engine JSON driver associated with the FTP account in the distributed database.

[0014] To achieve the above objectives, a second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the steps included in the above-mentioned method for collecting and processing massive power grid data.

[0015] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a massive power grid data acquisition and processing system, including an FTP server, a Netty server, multiple HES systems, a processor and the above-mentioned computer-readable storage medium; the Netty server includes a local KAFKA blocking queue, a Worker Group thread and a content parsing channel; The multiple HES systems are respectively connected to the FTP server, and the FTP server is connected to the Netty server; the processor is connected to the computer-readable storage medium and executes the computer program stored in the computer-readable storage medium.

[0016] In addition, the massive power grid data acquisition and processing system proposed in the above embodiment of the present invention may also have the following additional technical features: Optionally, it also includes a distributed database and an engine JSON driver associated with an FTP account; the distributed database is connected to the Netty server and the engine JSON driver respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the architecture of a massive power grid data acquisition and processing system proposed in an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a method for collecting and processing massive power grid data proposed in an embodiment of the present invention; Figure 3 A schematic diagram of a process for monitoring an FTP folder in a preferred specific implementation of an embodiment of the present invention; Figure 4 The figure is a flow chart of the Netty server reading and processing a newly added file in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0019] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0020] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] Explanation of the name: HES system: refers to the smart meter data acquisition system (HEAD-END SYSTEM) in the meter industry; HES tag: A field in the KAFKA message used to identify the data source HES system, which is usually an enumeration; FTP account: refers to the account and password used for authentication in the FTP (File Transfer Protocol) tool; Parsing rules: refers to the field-level data verification and parsing rules pre-configured for different CSV fields, that is, different FTP accounts; Netty server: server-side applications developed based on the Netty framework. Netty is an asynchronous event-driven network application framework and tools for rapid development of high-performance, high-reliability network servers and client programs; Shell script: A script program written using Shell commands, usually used to automate daily operating system tasks, such as file operations, program startup, etc. KAFKA blocking queue: In the Kafka message queue system, a queue used to store messages. It supports multiple producers and multiple consumers and can ensure the order of messages. The blocking queue here means that the thread will be blocked when the queue is empty or full; Content parsing channel: A channel used to process and parse data streams. It may be a module or component in the software that is responsible for converting input data into a usable format. Worker Group thread: In multithreaded programming, a collection of threads that usually perform the same or similar tasks and work together to complete a larger unit of work; Thread lock-free approach: A programming technique that reduces contention and synchronization overhead between threads by avoiding the use of locks to improve program performance and scalability; Singleton consumer thread: A thread with only one instance in the program, used to process a specific task or message queue to ensure that tasks are executed in order; Netty client: a client application developed based on the Netty framework, used to communicate with the Netty server over the network; Big data SPARK tasks: Big data processing tasks executed in the Spark distributed computing framework. Spark is used to process large-scale data sets and provide fast data analysis capabilities.

[0022] Figure 1 A schematic diagram of the architecture of a massive power grid data acquisition and processing system proposed in an embodiment of the present invention; Figure 2 The present invention provides a flowchart of a method for collecting and processing massive power grid data.

[0023] See also Figure 1 , an embodiment of the present invention provides a massive power grid data acquisition and processing system, such as Figure 1 As shown, it includes FTP server, Netty server, and multiple HES systems ( Figure 1 HES system a to HES system n in the FTP server); the Netty server includes a locally located KAFKA blocking queue, a Worker Group thread, and a content parsing channel; the multiple HES systems are respectively connected to the FTP server, and the FTP server is connected to the Netty server.

[0024] In this embodiment, the multiple HES systems correspond to third-party HES systems of different manufacturers. They can log in to the FTP server through their respective FTP accounts, and then upload files (such as CSV files) to access this system at a low cost to upload power grid data.

[0025] As a specific example, the HES systems of meter manufacturers A and B can log in to the same FTP server through their respective FTP accounts to access the system and report meter data of different ranges and types to the system. The different FTP accounts used by the HES systems of meter manufacturers A and B to log in to the same FTP server are obtained by the same data acquisition system, i.e., the system pre-configuration described in this embodiment. At the same time, the system will also generate the corresponding HES tags and user folders corresponding to different FTP accounts, i.e., FTP folders, based on the basic information and address of HES of A and B, respectively, to distinguish the data files of manufacturers A and B.

[0026] See also Figure 2 The embodiment of the present invention is based on Figure 1 The system architecture provides a method for collecting and processing massive power grid data, which specifically includes the following steps: S1: Pre-configure HES tags, FTP accounts, FTP folders and parsing rules corresponding to different HES systems; S2: Create mapping relationships between different HES systems and corresponding HES tags, FTP accounts, FTP folders, and parsing rules; Here, for the HES systems of different docking parties, by configuring the HES tags, FTP accounts, FTP folders and parsing rules that are uniquely corresponding to them and creating mapping relationships, accurate and efficient distinction and identification basis can be provided in the subsequent data collection and processing stages.

[0027] S3: Different HES systems log in to the same FTP server through their FTP accounts and upload their power grid data to the FTP folder corresponding to the FTP account in the FTP server; It can be understood that the HES data files of third-party HES systems of different manufacturers will be stored in different FTP folders with clear directionality; and based on the correspondence between HES system-FTP account-FTP folder in the mapping relationship, the correspondence between the HES system and the FTP folder can be accurately and efficiently locked.

[0028] S4: When a new file is detected in an FTP folder, a notification message carrying the corresponding HES tag is generated and sent to the Netty server; Here, the monitoring is performed by a listening program of the FTP server.

[0029] In some preferred embodiments, Figure 3As shown in the figure, the data collection layer of the system monitors the file changes in the FTP folder through the Shell script scheduled by the timed task; and when a new file is detected in a certain FTP folder, a notification message carrying the HES tag corresponding to the FTP folder is generated according to the mapping relationship, and then sent to the data parsing layer of the system through an HTTP request.

[0030] S5: The Netty server stores the notification message into the local KAFKA blocking queue; In some preferred specific implementations, after the Netty server located at the data parsing layer node receives the notification message sent by the FTP server listening program, it will identify the type of the notification message, and after confirming its validity, it will store the notification message in the local KAFKA blocking queue in a thread-free manner.

[0031] S6: the KAFKA blocking queue forwards the notification message to the content parsing channel, and reads the newly added file content from the FTP folder through the WorkerGroup thread, converts it into a file stream and sends it to the content parsing channel; In some preferred specific implementations, the singleton consumer thread in the local KAFKA blocking queue of the Netty server will read the notification messages in the queue in sequence, and then call the load balancing module to forward the notification messages to a Netty client for further processing. Here, the notification messages in the queue are processed based on load balancing using a distributed concurrent processing mechanism, which can ensure that the notification messages are processed efficiently and in a timely manner.

[0032] The Netty client that receives the notification message converts the notification message into a file read request corresponding to each newly added file according to the number of newly added files indicated in the notification message, and then distributes it to the Worker Group thread. That is, through the Netty client, a unique file read request is generated for each newly added file, and then distributed to the Worker Group thread pool.

[0033] The Worker Group thread that receives the file read request will read the file content from the corresponding newly added file in the FTP folder according to the request, and convert the read file content into a file stream and send it to the content parsing channel. Here, a file read request is handled by a Worker Group thread, and distributed concurrent processing is implemented through the Worker Group thread pool to maintain the efficiency of data reading. Optionally, the Worker Group thread reads the file content on the FTP server through the FTP file reading tool class.

[0034] S7: the content parsing channel parses the original power grid data in the file stream according to the mapping relationship and the parsing rule corresponding to the HES tag carried in the notification message, and converts it into standardized power grid data; Here, the content parsing channel located in the data processing layer of the system will parse the received file stream according to the HES tag carried in the notification message, by consulting the mapping relationship, and according to the pre-configured parsing rules corresponding to the HES tag. The parsing process includes specialized data cleaning, formatting and conversion, so as to convert the newly added file content into a unified format for subsequent analysis and storage.

[0035] S8: Store standardized power grid data in a distributed database.

[0036] In some preferred specific implementations, the parsing results of the content parsing channel, i.e., the standardized data, will be sent to the KAFKA exclusive queue; after the KAFKA exclusive queue performs streaming calculations on the standardized power grid data through the big data SPARK task, the calculation results are stored in the distributed database Apache Doris located at the system application layer. Here, by uniformly transferring and storing massive standardized power grid data into the distributed database, the data query efficiency can be improved, while the storage cost can be reduced, and the needs of power data analysis can be better met.

[0037] In some specific implementations, the process of the Netty server reading the newly added file and processing it is as follows Figure 4 As shown, the file download is performed through the Worker Group thread, and the file parsing is performed through the content parsing channel. Here, timely feedback processing will be performed for unsuccessful downloads, and feedback and backup will be performed even if the download is successful. It can be seen that this embodiment has a complete and rigorous data processing logic, and the system has strong robustness.

[0038] In some further preferred embodiments of this embodiment, after the step S8, the following steps are further included: S9: Access the engine JSON driver associated with the FTP account in the distributed database.

[0039] Here, the data uploaded by different HES systems may have different application requirements. For example, Manufacturer A hopes that the uploaded meter data will be converted and then passed to the third-party platform through the interface as data support for the application system; Manufacturer B hopes that the uploaded meter data will be stored according to the agreed dimensions and support high-performance data analysis and query basis. Therefore, after the data processing flow, different driver processing can be performed according to the process engine JSON pre-configured in this system according to different HES systems to meet different data application requirements.

[0040] A method for collecting and processing massive power grid data provided in this embodiment is based on technologies such as Netty, Apache Kafka, Java, Apache SPARK, FTP, etc., and provides an advanced and efficient massive power grid data reading solution for the power grid MDM system. It has a targeted data processing method with characteristics of the electric meter industry, allows multi-connection parties and multi-connection formats of power grid reporting data processing, can significantly improve the efficiency and reliability of the system, provides practical innovative technology for the smart grid field, and provides a solid technical foundation for various derivative applications based on the smart grid advanced measurement system architecture.

[0041] The embodiment of the present invention is further extended based on the above embodiment, and also provides a computer-readable storage medium.

[0042] The computer-readable storage medium provided in this embodiment stores a computer program, which, when executed by a processor, can implement the steps of a method for collecting and processing massive power grid data described in the above embodiment. The specific steps are not repeated here, and please refer to the records of the above embodiment for details.

[0043] The embodiment of the present invention is further expanded based on the above embodiment, and also provides a massive power grid data acquisition and processing system.

[0044] The massive power grid data acquisition and processing system provided in this embodiment includes: Figure 1 The FTP server, Netty server, multiple HES systems and the computer-readable storage medium described in the above embodiment ( Figure 1 The Netty server includes a local KAFKA blocking queue, a Worker Group thread, and a content parsing channel; The multiple HES systems are respectively connected to the FTP server, and the FTP server is connected to the Netty server; the processor is connected to the computer-readable storage medium and executes the computer program stored in the computer-readable storage medium.

[0045] In some preferred specific implementations, the massive power grid data acquisition and processing system also includes a distributed database and an engine JSON driver associated with an FTP account; the distributed database is connected to the Netty server and the engine JSON driver respectively.

[0046] The system of this embodiment can realize real-time synchronous docking of multiple HES systems, high-frequency data reception and processing. For example: the file data reported by Manufacturer A and Manufacturer B through their respective HES systems have certain differences in format, data items and data range. The system of this embodiment is based on Netty's distributed data collection layer and mapping relationship to distinguish and classify the collected data, corresponding to the HES system (i.e., different manufacturers) in the system configuration; the reading and preprocessing of data as a whole follows the distributed process concurrent processing mechanism based on Netty to ensure the efficiency and frequency of data collection, and in the process of data cleaning, formatting and conversion, the data of different HES manufacturers are differentiated and preprocessed according to the corresponding strategy mode (parsing rules), so that the data uploaded by Manufacturer A and the data uploaded by Manufacturer B are respectively subjected to different data cleanings adapted thereto to obtain different standard results, ensuring the availability and effectiveness of the data in the subsequent storage layer and application layer.

[0047] The system of this embodiment can support multi-link downstream processing of pre-processed data, including distributed storage and third-party message sending, has flexible availability and high support for complex system architecture design, has good scalability and fault tolerance, and can quickly respond to abnormal situations. For example: different manufacturers may have different application requirements for data uploaded through the HES system. For example, Manufacturer A hopes that the uploaded meter data will be converted and then passed to the third-party platform through the interface as data support for the application system; Manufacturer B hopes that the uploaded meter data will be stored according to the agreed dimensions and support high-performance data analysis and query basis. Therefore, after the data processing flow, different drive processing can be performed according to the process engine JSON configured by Manufacturer A and Manufacturer B in the system.

[0048] It can be seen that this embodiment can construct a high-throughput, scalable, highly reliable, and standard storage power grid data acquisition and processing system to better cope with the growing scale of power grid data and build a general power grid advanced measurement architecture. The massive power grid data acquisition and processing system of this embodiment can be an electric meter MDM system, which is based on the non-blocking communication protocol model Netty and the message middleware Kafka, and can efficiently and accurately process the power grid reported data with the characteristics of the electric meter industry, and allow different docking parties and different docking formats to report power grid data. In addition, the system of this embodiment makes full use of the distributed technology framework, which can significantly improve the timeliness, accuracy, reliability and security of data extraction.

[0049] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0053] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0054] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0055] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0056] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0057] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0059] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily being directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0060] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for collecting and processing massive power grid data, characterized in that: include: Pre-configure HES tags, FTP accounts, FTP folders and parsing rules corresponding to different HES systems; Create mapping relationships between different HES systems and corresponding HES tags, FTP accounts, FTP folders, and parsing rules; Different HES systems upload their power grid data to the corresponding FTP folders through their FTP accounts; When a new file is detected in an FTP folder, a notification message with the corresponding HES tag is generated and sent to the Netty server. The Netty server stores the notification message in the local KAFKA blocking queue; The KAFKA blocking queue forwards the notification message to the content parsing channel, and reads the newly added file content from the FTP folder through the Worker Group thread, and converts it into a file stream and sends it to the content parsing channel; The content parsing channel parses the original power grid data in the file stream according to the mapping relationship and the parsing rule corresponding to the HES tag carried in the notification message, and converts it into standardized power grid data; Store standardized power grid data in a distributed database.

2. A method for collecting and processing massive power grid data according to claim 1, characterized in that: The changes of the FTP folder are monitored through a Shell script.

3. A method for collecting and processing massive power grid data according to claim 1, characterized in that: The Netty server stores the notification message into the local KAFKA blocking queue in a thread-free manner.

4. A method for collecting and processing massive power grid data according to claim 1, characterized in that: The reading of the newly added file content from the FTP folder through the WorkerGroup thread and converting it into a file stream and sending it to the content parsing channel includes: The singleton consumer thread in the KAFKA blocking queue reads the notification message and calls the load balancing module to forward the notification message to a Netty client; The Netty client converts the notification message into a file read request corresponding to each newly added file and distributes it to the Worker Group thread; The Worker Group thread that receives the file read request reads the corresponding newly added file content in the FTP folder, and converts it into a file stream and sends it to the content parsing channel.

5. A method for collecting and processing massive power grid data according to claim 1, characterized in that: The analysis process includes data cleaning and formatting.

6. A method for collecting and processing massive power grid data according to claim 1, characterized in that: The storing of standardized power grid data in a distributed database includes: Sending the standardized power grid data to the KAFKA exclusive queue; The KAFKA exclusive queue performs streaming calculations on the standardized power grid data through the big data SPARK task, and stores the calculation results in a distributed database.

7. A method for collecting and processing massive power grid data according to claim 1, characterized in that: Also includes: Access the engine JSON driver associated with the FTP account in the distributed database.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the program can implement the steps included in the method for collecting and processing massive power grid data as described in any one of claims 1 to 7.

9. A massive power grid data acquisition and processing system, characterized in that: It includes an FTP server, a Netty server, a plurality of HES systems, a processor, and the computer-readable storage medium described in claim 8; the Netty server includes a locally located KAFKA blocking queue, a Worker Group thread, and a content parsing channel; The multiple HES systems are respectively connected to the FTP server, and the FTP server is connected to the Netty server; the processor is connected to the computer-readable storage medium and executes the computer program stored in the computer-readable storage medium.

10. A massive power grid data acquisition and processing system as claimed in claim 9, characterized in that: It also includes a distributed database and an engine JSON driver associated with an FTP account; the distributed database is connected to the Netty server and the engine JSON driver respectively.

Images