Method and system for quickly encapsulating and sharing data of power system, medium and processor

Abstract

The invention relates to an electric power system data rapid packaging and sharing method, which specifically comprises the following steps of: connecting with each data source for storing electric power data, and collecting the electric power data to form original data; carrying out convergence, processing and fusion processing on the business power data in the original data to form basic data; classifying and organizing the basic data, establishing a data index, packaging the data of different data sources through parameter configuration, generating a data packet, storing the data packet on a local directory of a server or a distributed server, and forming a data product for a user to access and use; and establishing a unified data visualization view, receiving various types of data access requests of the user, and then calling each professional data background to provide data product access service to the outside. A data access technology for shielding bottom layer data diversity is researched aiming at a business scene, a unified service interface definition is provided for data source support of the same type, and the familiarity process and the work complexity of a user for docking each data source are simplified.

Description

Technical Field

[0001] The present invention relates to the technical field of power data interaction, and in particular to a method, system, medium and processor for rapid encapsulation and sharing of power system data. Background Art

[0002] With the continuous development of energy facilities and technologies, smart grid has become one of the main development directions of future energy systems. Smart grid is a power system with information technology as its core. It has completely upgraded and transformed all aspects of power transmission, distribution and power supply in traditional power grids, making the entire power grid system smarter, more efficient, safer and more reliable.

[0003] In response to the large-scale consumption of new energy and to cope with the challenges brought by the construction of new power systems, the number and scale of various intelligent business scenarios of power systems are increasing, involving the development and training of a large number of AI models such as intelligent regulation, operation error prevention, and new energy prediction. In the process, a massive amount of detection, control, and shared power data will need to be processed.

[0004] As the number and scale of various intelligent business scenarios in the power system continue to increase, the cross-business demands between systems are also beginning to increase. The familiarization process that originally relied on users to manually connect to each data source is becoming more and more cumbersome, and the complexity of the work and the time and labor costs are increasing sharply. At present, there is no corresponding method to realize the automation of data interaction and sharing operations between various intelligent business scenarios in the power industry.

[0005] In view of this, a method, system, medium and processor for rapid encapsulation and sharing of power system data are needed. Summary of the invention

[0006] In view of the problem that there is no corresponding method in the prior art to realize an automated solution for data interactive sharing operations between various intelligent business scenarios of the power industry, the present invention provides a method, system, medium and processor for rapid encapsulation and sharing of power system data, which can realize data encapsulation and interactive sharing between various intelligent business scenarios of the power industry. The specific technical solution is as follows:

[0007] A method for quickly encapsulating and sharing power system data, specifically comprising the following steps:

[0008] Connect with various data sources storing power data, collect power data and assign timestamps, and generate raw data after compliance and integrity checks;

[0009] Aggregate, process and fuse the power data of each business in the original data to form basic data;

[0010] Classify and organize basic data, establish data indexes, package data from different data sources through parameter configuration, generate data packages and store them in the local directory of the server or on distributed servers to form data products for users to access and use;

[0011] Establish a unified data visualization view, provide data directory, information display, retrieval and download, subscription and distribution functions, receive various types of data access requests from users, and then call various professional data backends to provide external data product access services.

[0012] Furthermore, the original data is stored and processed using different data storage frameworks based on the diversity of power data and for different data structures.

[0013] Furthermore, the storage processing using different data storage frameworks for different data structures includes the following steps:

[0014] For the processing of structured data, a combination of relational database and spatial database is used to achieve high efficiency, real-time and security of data storage and access;

[0015] The processing of unstructured data uses NoSQL database and object storage to achieve efficient, real-time and secure data storage and access.

[0016] Furthermore, each data source includes various data centers, data warehouses or cloud platforms.

[0017] Furthermore, when receiving multiple types of data access requests from users, different users are granted different data access control permissions; the data access control permissions are finely granularly set according to data classification; the permission range includes invisible, browsable and downloadable, and the corresponding time limit range includes long-term, short-term and temporary.

[0018] Furthermore, the aggregation, processing and fusion processing, including data cleaning and conversion, data verification and integration, data fusion and other process operations, greatly improves the quality and accuracy of the data.

[0019] Furthermore, the compliance and integrity check refers to periodic data set quality checks based on the data update frequency to ensure data quality, specifically including the following check steps:

[0020] Incremental data check to ensure that new data is generated in each cycle;

[0021] Check data fluctuations to ensure that incremental data is within a reasonable range;

[0022] Check the null value rate of key fields to ensure that the null value rate of key fields is within a reasonable range;

[0023] Check the uniqueness of the primary key or composite primary key to ensure that no duplicate dirty data is generated.

[0024] A power system data rapid encapsulation and sharing system, applied to the power system data rapid encapsulation and sharing method described above, comprising:

[0025] A data acquisition module is used to connect to various data sources storing power data, collect power data and assign timestamps, and form raw data after compliance and integrity checks;

[0026] A data processing module is used to aggregate, process and fuse the power data of each business in the original data to form basic data;

[0027] The data encapsulation module is used to classify and organize basic data, establish data indexes, package data from different data sources through parameter configuration, generate data packets and store them in the local directory of the server or on the distributed server to form data products for users to access and use;

[0028] The data sharing module is used to establish a unified data visualization view, provide data directory, information display, retrieval and download, subscription and distribution functions, receive various types of data access requests from users, and then call various professional data backends to provide external data product access services.

[0029] A computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for rapid encapsulation and sharing of power system data.

[0030] A processor is used to run a program, wherein the program executes the above-mentioned method for rapid encapsulation and sharing of power system data when running.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] Aiming at the cognitive, decision-making and prediction business scenarios in the power field, we study data access technology that shields the diversity of underlying data, provide a unified service interface definition for the same type of data sources, and implement differentiated requests for various types of middleware on the platform, simplifying the user's familiarity process and work complexity in connecting to each data source. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0034] Figure 1 A flowchart of a method for rapid encapsulation and sharing of power system data is provided;

[0035] Figure 2 The schematic diagram is a structural diagram of a power system data rapid encapsulation and sharing system. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0038] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0039] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] Embodiment 1

[0041] like Figure 1 The figure shows a flow diagram of a method for rapid encapsulation and sharing of power system data. In the whole method flow, the storage partition is divided into an original data area, a basic data area, a service product area and a common support area; the method specifically includes the following steps:

[0042] S1: Connect with various data sources storing power data, collect power data and assign timestamps, perform compliance and integrity checks, and generate raw data to be stored in the raw data area.

[0043] Furthermore, each data source includes various data centers, data warehouses or cloud platforms. That is, the original data area mainly stores various external observation and detection power status data, power equipment data and various intelligent business scenario data. The original data area does not modify the data, but only performs compliance and integrity checks; a unified timestamp is given when the original data is collected to mark the time sequence of data collection.

[0044] Furthermore, based on the diversity of power data, different data storage frameworks are used in the original data area to store and process the original data for different data structures.

[0045] Furthermore, the storage processing using different data storage frameworks for different data structures includes the following steps:

[0046] For the processing of structured data, a combination of relational database and spatial database is used to achieve high efficiency, real-time and security of data storage and access;

[0047] The processing of unstructured data uses NoSQL database and object storage to achieve efficient, real-time and secure data storage and access.

[0048] Furthermore, the compliance and integrity check refers to periodic data set quality checks based on the data update frequency to ensure data quality, specifically including the following check steps:

[0049] Incremental data check to ensure that new data is generated in each cycle;

[0050] Check data fluctuations to ensure that incremental data is within a reasonable range;

[0051] Check the null value rate of key fields to ensure that the null value rate of key fields is within a reasonable range;

[0052] Check the uniqueness of the primary key or composite primary key to ensure that no duplicate dirty data is generated.

[0053] Furthermore, a timestamp is assigned when collecting power data to mark the collection time of the data, so as to facilitate the analysis of the changing trends of the power distribution and power supply status in each area.

[0054] Furthermore, based on the vastness of the area covered by the power system, meteorological thresholds are set to monitor the meteorological conditions of each region, and the power data of the extreme weather areas are updated in real time. At the same time, a curve can be fitted to the change of the power data, and the curve can be derived to obtain the degree of change of the power data over time. The frequency of collection can be reduced for slowly changing power data, and the frequency of collection can be increased for drastically changing power data, so as to provide the accuracy of shared data reflecting the state of the power grid. Although the collection frequency of each region is different, the time of collection can be set uniformly.

[0055] S2: Aggregate, process and fuse the business power data in the original data to form basic data storage in the basic data area.

[0056] Furthermore, the aggregation, processing and fusion processing, including data cleaning and conversion, data verification and integration, data fusion and other process operations, greatly improves the quality and accuracy of the data.

[0057] Furthermore, the aggregation, processing and fusion processing also includes sequentially transcoding, format conversion and quality inspection operations on the data.

[0058] Data fusion and processing refers to the collection, integration, analysis and processing of data from different sources and different types to achieve the purpose of improving data accuracy, completeness and practicality. In smart grids, data fusion and processing technology plays a vital role. Because the data sources involved in smart grids are very wide, including the telemetry, remote control, remote signaling and other system data of the power grid itself, as well as user electrical appliances and environmental data, and even multiple external environmental data sources such as weather forecasts and economic data.

[0059] The aggregation, processing and fusion process comprises the following steps:

[0060] Data collection and storage, data collection and storage is the basis of data fusion and processing. It can collect, store, archive and manage data to ensure the accuracy, completeness and practicality of the data.

[0061] Data preprocessing refers to cleaning, filling missing values, processing outliers, and reducing noise before data fusion and processing. Preprocessing can effectively improve the quality of data and ensure the accuracy and effectiveness of subsequent operations.

[0062] Data fusion is the process of integrating and matching data from different data sources to form a consistent data structure and format. This facilitates unified analysis and modeling of multi-source data and provides basic support for subsequent data processing.

[0063] Data mining and intelligent analysis,Data mining and intelligent analysis refer to the in-depth analysis and modeling of data, extracting useful information and knowledge from the data to provide support for business planning, decision making and operation optimization.

[0064] S3: Classify and organize basic data, such as reorganizing basic data according to the needs of various professional fields, establishing data indexes, packaging and processing data from different data sources through parameter configuration, generating data packets and storing them in the local directory of the server or on the distributed server, that is, storing them in the service product area, forming data products for users to access and use. That is, the service product area is responsible for providing external data product services, mainly storing product data processed by service products in various professional fields in the system.

[0065] In a specific implementation, the data from different data sources are packaged and processed through parameter configuration, including the following steps:

[0066] Determine the type and attributes of the data. Before data encapsulation, you first need to determine the type and attributes of the data. The data type can be a basic type, such as an integer, a floating point number, a string, or a custom type, such as a structure, a class, etc. The attributes of the data are information that describes the characteristics and status of the data. By determining the type and attributes of the data, you can clarify how the data is stored and how it is accessed, laying the foundation for subsequent data encapsulation.

[0067] Design data interfaces and methods. Data interfaces refer to a set of methods or functions used to access and manipulate data. When designing data interfaces, you need to consider the usage scenarios and requirements of the data and follow the principle of encapsulation, that is, hide the internal implementation details of the data and only expose the necessary interfaces for external use. The interface design should be concise, provide necessary functions, and follow certain naming conventions and parameter conventions.

[0068] Data encapsulation means encapsulating data and data-related methods together according to the data type and attributes. When implementing data encapsulation, appropriate data structures and algorithms need to be used to improve data access efficiency and operation efficiency. At the same time, data security and integrity need to be considered, and the legitimacy of data needs to be protected through access control and data verification.

[0069] Test data encapsulation. Testing is one of the key steps to ensure the quality of data encapsulation. When testing data encapsulation, you need to write corresponding test cases and conduct comprehensive tests on the data interfaces and methods to ensure the correctness and stability of their functions. At the same time, you also need to consider boundary conditions and abnormal situations to improve the robustness and fault tolerance of the data.

[0070] Optimize data encapsulation. Optimization is a continuous improvement process of data encapsulation. When optimizing data encapsulation, you can consider using more efficient data structures and algorithms, improving the design and implementation of interfaces, reducing data storage and access overhead, and improving data performance and scalability. At the same time, you can also adjust and expand data encapsulation according to actual needs to meet different business needs.

[0071] Data encapsulation is a technology that encapsulates data and operations on data. By determining the type and attributes of data, designing data interfaces and methods, implementing data encapsulation, testing data encapsulation, optimizing data encapsulation and other steps, secure access to and operations on data can be achieved. Data encapsulation can not only improve the security and integrity of data, but also provide a concise interface for other programs or modules to use, thereby improving the reusability and maintainability of the code. Therefore, mastering the technology of data encapsulation is a basic skill that every programmer must have.

[0072] S4: Establish a unified data visualization view, provide data directory, information display, retrieval and download, subscription and distribution functions, receive various types of data access requests from users, and then call various professional data backends to provide external data product access services.

[0073] Furthermore, when receiving multiple types of data access requests from users, different data access control permissions are granted to different users; the data access control permissions are finely granularly set according to data classification; the permission range includes invisible, browsable and downloadable, and the corresponding time limit range includes long-term, short-term and temporary.

[0074] At the same time, different data access control permissions and data sharing access interface permissions are granted to different users to complete cross-system and cross-industry data sharing, provide shared access to different professional data, and realize a transparent data access process for users.

[0075] When the data access control permissions are finely granularly set according to data classification, the implementation of fine-grained data access control requires consideration of data sensitivity and user permissions. By combining identity authentication, authorization management, audit logs, data encryption and other measures, it can be ensured that only authorized users can access sensitive data, and that user access behavior is recorded to prevent unauthorized access. Specifically, the following steps are included:

[0076] First, identity verification is the first step to ensure that only authorized users can access data. This is usually done through username and password, fingerprint recognition, face recognition, etc. After identity verification, the system will further check the user's permissions to ensure that they can only access authorized data.

[0077] Secondly, authorization management is the key to controlling user access rights. Through role management, permission management, access control list (ACL), etc., it can be ensured that users can only access data for which they have permission. For example, in Java, access rights modifiers such as public, protected, and private can be used to control the visibility of classes, methods, and variables.

[0078] Audit logs are used to monitor and record user access behaviors, and to promptly detect and handle abnormal access behaviors. Audit logs can record user access time, access content, access results and other information, which helps track and investigate potential security issues.

[0079] Data encryption is an important means of protecting sensitive data. By encrypting the storage and transmission of sensitive data, unauthorized access can be prevented. In Java, built-in encryption libraries such as Cipher and MessageDigest can be used to implement data encryption.

[0080] Assume there is an employees table, you can create a policy function to restrict users to only see the employee records of their department. This is achieved through a PL / SQL function that returns a WHERE clause that restricts users to only see the employee records of their department.

[0081] Finally, data sensitivity is also an important factor in achieving fine-grained control. Data sensitivity refers not only to the confidentiality level of data, but also to people's keen perception of data. By improving users' data sensitivity and business understanding, data can be better understood and used, thereby improving the effectiveness of data access control.

[0082] In addition to the above functions, there is also a shared support area, which mainly stores the supporting environment data required for the operation of the business system, mainly including metadata and system operation and maintenance management data.

[0083] Embodiment 2

[0084] A power system data rapid encapsulation and sharing system, applied to the power system data rapid encapsulation and sharing method described above, comprising:

[0085] A data acquisition module is used to connect to various data sources storing power data, collect power data and assign timestamps, and form raw data after compliance and integrity checks;

[0086] A data processing module is used to aggregate, process and fuse the power data of each business in the original data to form basic data;

[0087] The data encapsulation module is used to classify and organize basic data, establish data indexes, package data from different data sources through parameter configuration, generate data packets and store them in the local directory of the server or on the distributed server to form data products for users to access and use;

[0088] The data sharing module is used to establish a unified data visualization view, provide data directory, information display, retrieval and download, subscription and distribution functions, receive various types of data access requests from users, and then call various professional data backends to provide external data product access services.

[0089] Embodiment 3

[0090] A computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for rapid encapsulation and sharing of power system data.

[0091] Embodiment 4

[0092] A processor is used to run a program, wherein the program executes the above-mentioned method for rapid encapsulation and sharing of power system data when running.

[0093] The present application discloses a method for rapid encapsulation and sharing of power system data, which specifically includes the following steps: connecting with various data sources storing power data, collecting power data and assigning timestamps, and performing compliance and integrity checks to form raw data; aggregating, processing and fusing various business power data in the raw data to form basic data; classifying and organizing the basic data, establishing data indexes, packaging and processing the data from different data sources through parameter configuration, generating data packets and storing them in the local directory of the server or on the distributed server, and forming data products for users to access and use; establishing a unified data visualization view, providing data directory, information display, retrieval and download, subscription and distribution functions, receiving various types of data access requests from users, and then calling various professional data backends to provide data product access services to the outside world. Aiming at the cognitive, decision-making and prediction business scenarios in the power field, the data access technology that shields the diversity of underlying data is studied, and a unified service interface definition is provided for the same type of data source support. The platform implements differentiated requests of various middleware, which simplifies the familiarization process and work complexity of users connecting to each data source.

[0094] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0095] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0096] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for rapid encapsulation and sharing of power system data, characterized in that: The specific steps include: Connect with various data sources storing power data, collect power data and assign timestamps, and generate raw data after compliance and integrity checks; Aggregate, process and fuse the power data of each business in the original data to form basic data; Classify and organize basic data, establish data indexes, package data from different data sources through parameter configuration, generate data packages and store them in the local directory of the server or on distributed servers to form data products for users to access and use; Establish a unified data visualization view, provide data directory, information display, retrieval and download, subscription and distribution functions, receive various types of data access requests from users, and then call various professional data backends to provide external data product access services.

2. The method for rapid encapsulation and sharing of power system data according to claim 1, characterized in that: The raw data is based on the diversity of power data and adopts different data storage frameworks for storage and processing according to different data structures.

3. The method for rapid encapsulation and sharing of power system data according to claim 2, characterized in that: The storage process using different data storage frameworks for different data structures includes the following steps: For the processing of structured data, a combination of relational database and spatial database is used to achieve high efficiency, real-time and security of data storage and access; The processing of unstructured data uses NoSQL database and object storage to achieve efficient, real-time and secure data storage and access.

4. The method for rapid encapsulation and sharing of power system data according to claim 1, characterized in that: The data sources include various data centers, data warehouses or cloud platforms.

5. The method for rapid encapsulation and sharing of power system data according to claim 1, characterized in that: When receiving multiple types of data access requests from users, different users are granted different data access control permissions; the data access control permissions are finely granularly set according to data classification; the permission range includes invisible, browsable and downloadable, and the corresponding time limit range includes long-term, short-term and temporary.

6. The method for rapid encapsulation and sharing of power system data according to claim 1, characterized in that: The aggregation, processing and fusion processing include data cleaning and conversion, data verification and integration, data fusion and other process operations, which greatly improves the quality and accuracy of the data.

7. The method for rapid encapsulation and sharing of power system data according to claim 1, characterized in that: The compliance and integrity check refers to the periodic data set quality check based on the data update frequency to ensure data quality, which specifically includes the following check steps: Incremental data check to ensure that new data is generated in each cycle; Check data fluctuations to ensure that incremental data is within a reasonable range; Check the null value rate of key fields to ensure that the null value rate of key fields is within a reasonable range; Check the uniqueness of the primary key or composite primary key to ensure that no duplicate dirty data is generated.

8. A power system data rapid packaging and sharing system, characterized in that: The method for rapid encapsulation and sharing of power system data applied to any one of claims 1 to 7 comprises: A data acquisition module is used to connect to various data sources storing power data, collect power data and assign timestamps, and form raw data after compliance and integrity checks; A data processing module is used to aggregate, process and fuse the power data of each business in the original data to form basic data; The data encapsulation module is used to classify and organize basic data, establish data indexes, package data from different data sources through parameter configuration, generate data packets and store them in the local directory of the server or on the distributed server to form data products for users to access and use; The data sharing module is used to establish a unified data visualization view, provide data directory, information display, retrieval and download, subscription and distribution functions, receive various types of data access requests from users, and then call various professional data backends to provide external data product access services.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for rapid encapsulation and sharing of power system data as described in any one of claims 1 to 7.

10. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the method for rapid encapsulation and sharing of power system data as described in any one of claims 1 to 7.

Images