Big data processing and analysis system based on electrical energy measurement data

Abstract

The invention discloses a big data processing and analysis system based on electric power energy measurement data, and belongs to the technical field of data processing and analysis in the electric power industry. The system comprises a data management service module, a model service module, a self-service analysis module, an exploration analysis module and a data application management module. The data management service model is used for managing system services, the model service module is used for designing an algorithm model and managing the designed algorithm model, and the self-service analysis module is used for automatically generating an SQL based on electric energy measurement data. The exploration analysis module is used for performing data processing on the electrical energy measurement data by using an algorithm model designed by the model service module to generate a data processing result, and the data application management module is used for managing the electrical energy measurement data. According to the invention, the data acquisition efficiency and quality are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and analysis in the electric power industry, and more specifically, to a big data processing and analysis system based on electric power energy metering data. Background Art

[0002] With the continuous accumulation of metering data in the power industry, how to efficiently process this data has become an important issue. At present, although the metering center has gathered a large amount of data, the acquisition, analysis and application of data still face challenges. Business personnel usually find it difficult to directly obtain the required data, and cannot conduct in-depth data analysis on their own. In addition, data analysis work often relies on professional data developers, which limits the efficiency and flexibility of data use. Summary of the invention

[0003] In view of the above problems, the present invention proposes a big data processing and analysis system based on electric energy metering data, including: a data management service module, a model service module, a self-service analysis module, an exploration analysis module and a data application management module;

[0004] The data management service model is used to manage system services, the model service module is used to design algorithm models and manage the designed algorithm models, the self-service analysis module is used to automatically generate SQL based on power energy metering data, the exploration analysis module is used to use the algorithm model designed by the model service module to process the power energy metering data and generate data processing results, and the data application management module is used to manage the power energy metering data.

[0005] Optionally, the data management service model is used to manage system services, including:

[0006] Service publishing, service subscription, service authorization and service review.

[0007] Optionally, the model service module is also used to manage the calling of algorithm models.

[0008] Optionally, the self-service analysis module automatically generates SQL through graphical drag, drop and drop.

[0009] Optionally, the exploratory analysis module is specifically used to:

[0010] Perform data calculation and processing, data mining and model application on electric energy metering data, and display the data in graphical form.

[0011] Optionally, the data application management module is specifically used to display charts and manage the power energy metering data using a drawing board.

[0012] Optional, electrical energy metering data, including:

[0013] General metering data of electric energy.

[0014] Optionally, the exploration and analysis module is also used to sort out the scope of general support data requirements and build a general data sample market for measurement data based on typical scenario application research requirements;

[0015] The data sample sets included in the general data sample market of metering data include: metering user archive data sample set, metering equipment archive data sample set, metering equipment full life cycle data sample set, power user daily electricity data sample set and power user load curve data sample set.

[0016] Optionally, the exploration and analysis module is also used to manage sample set data, including:

[0017] Retrieve sample data subsets, view sampling rules, browse sample data, set the accessible scope of specific sample data sets, and take sample data sets offline.

[0018] Optionally, the system further includes: an index release management tool for:

[0019] Create new analytical indices based on existing indicators, basic data and business rules.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention provides a big data processing and analysis system based on electric energy metering data, comprising: a data management service module, a model service module, a self-service analysis module, an exploration analysis module and a data application management module; the data management service model is used to manage system services, the model service module is used to design an algorithm model and manage the designed algorithm model, the self-service analysis module is used to automatically generate SQL based on electric energy metering data, the exploration analysis module is used to process the electric energy metering data using the algorithm model designed by the model service module to generate data processing results, and the data application management module is used to manage the electric energy metering data. The present invention improves the efficiency and quality of data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of the system of the present invention;

[0023] Figure 2 It is a framework diagram of the system of the present invention;

[0024] Figure 3 A content framework diagram for constructing data processing and analysis tools for the system of the present invention. DETAILED DESCRIPTION

[0025] Now, exemplary embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely and to fully convey the scope of the present invention to those skilled in the art. The terms used in the exemplary embodiments shown in the accompanying drawings are not intended to limit the present invention. In the accompanying drawings, the same units / elements are marked with the same reference numerals.

[0026] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.

[0027] The present invention proposes a big data processing and analysis system based on electric energy metering data, such as Figure 1 As shown, it includes: data management service module, model service module, self-service analysis module, exploration analysis module and data application management module;

[0028] The data management service model is used to manage system services, the model service module is used to design algorithm models and manage the designed algorithm models, the self-service analysis module is used to automatically generate SQL based on power energy metering data, the exploration analysis module is used to use the algorithm model designed by the model service module to process the power energy metering data and generate data processing results, and the data application management module is used to manage the power energy metering data.

[0029] The data management service model is used to manage system services, including:

[0030] Service publishing, service subscription, service authorization and service review.

[0031] Optionally, the model service module is also used to manage the calling of algorithm models.

[0032] Among them, the self-service analysis module automatically generates SQL through graphical drag, drop and drop.

[0033] The exploration and analysis module is specifically used for:

[0034] Perform data calculation and processing, data mining and model application on electric energy metering data, and display the data in graphical form.

[0035] Among them, the data application management module is specifically used to display charts and manage the power energy metering data with a drawing board.

[0036] Among them, power energy metering data includes:

[0037] General metering data of electric energy.

[0038] Among them, the exploration and analysis module is also used to sort out the scope of general support data requirements and build a general data sample market for measurement data based on the research needs of typical scenario applications;

[0039] The data sample sets included in the general data sample market of metering data include: metering user archive data sample set, metering equipment archive data sample set, metering equipment full life cycle data sample set, power user daily electricity data sample set and power user load curve data sample set.

[0040] The exploration and analysis module is also used to manage sample set data, including:

[0041] Retrieve sample data subsets, view sampling rules, browse sample data, set the accessible scope of specific sample data sets, and take sample data sets offline.

[0042] The system also includes: index release management tools, which are used to:

[0043] Create new analytical indices based on existing indicators, basic data and business rules.

[0044] The system framework is as follows: Figure 2 As shown, it includes: data management services, model services, self-service analysis capabilities, data application management modules, and supports graphical data analysis suites. Users do not need to master SQL, data processing and other skills. Through graphical dragging, clicking drill-down and other operations, they can generate the required analysis conclusions. The core capabilities are described in detail as follows.

[0045] General data set for power metering. Based on the research needs of typical scenarios, the scope of general support data needs is sorted out, and a general data sample market for metering data is built for regular data analysis and model training. The following sample sets are established in this issue: metering user archive data sample set, metering equipment archive data sample set, metering equipment full life cycle data sample set, power user daily electricity data sample set, and power user load curve data sample set.

[0046] Sub-topic data sampling. For data sets with large original data, different sampling methods and rules (with replacement, without replacement, and proportional sampling methods) can be used as needed to screen data sample subsets and produce a small-scale sample set of the original data set to meet the needs of data analysts for rapid verification, immediate analysis and query, and other analysis scenarios. When using proportional data sampling, a single-factor or multi-factor stratified sampling mode can be set according to actual needs, so that the sample data set retains some statistical characteristics of the original data set.

[0047] Sample data management: Sample data management implements the management functions of the established sample data sets, including the retrieval of sample data subsets, sampling rule viewing, sample data browsing, setting the accessible scope of specific sample data sets, and offline of sample data sets.

[0048] Power metering analysis sandbox: To support business analysts to use power metering general sample data sets to conduct data exploration and analysis, model training, and idea verification, and to isolate it from the business production system environment and data, a power metering analysis sandbox environment is built to support the operation environment of data processing, data exploration and analysis, and multi-dimensional statistical analysis activities based on general sample data subsets.

[0049] Data intelligent analysis tools. Data intelligent analysis tools provide a platform analysis environment to support business personnel in conducting regular business analysis. Based on the general sample data set of power metering, data exploration analysis and model construction are realized, including: cross-dimensional jump drilling, reuse of analysis paths, comparative analysis, model construction, multi-person collaboration, etc., to help users quickly gain insights into data patterns and rules, so that business personnel can carry out analysis work without IT support personnel, help users quickly build enterprise analysis capabilities, share data analysis results, and promote the transformation process from business-driven to data-driven.

[0050] Index release management tool. The index release management tool provides business personnel with tools to create new analytical indexes based on existing indicators and basic data, according to business rules and needs, and provides tools for index management. Index creation mainly uses data tables that have been cleaned and standardized. After supporting data exploration and screening, supporting data value transformation, indicator system construction, and setting fixed operations, index creation is achieved. According to business needs, early warning rules can be set for specific indexes to push information.

[0051] Data application creation tool. The data application creation tool allows users to use the data charts created in the intelligent analysis tool to create a publishable data application page according to the specified layout structure. Users can set page filtering conditions for each chart and add page and column titles, introductions, icons and other decorative materials. Users can browse the data application pages they created, share the created pages with designated users for browsing, or submit the created pages to the data application warehouse for review.

[0052] Data application management. Data application management provides management functions for data application pages created by users, provides data application retrieval, editing, and viewing functions, and can set the sharing scope of the application (only visible to oneself, visible to specific users, fully public) and delete data application pages during application release.

[0053] The present invention can significantly improve the processing and analysis efficiency of metering data, enhance the degree of self-service of data analysis, and reduce the dependence on professional data developers, thereby improving the flexibility and response speed of the entire system. Through the present invention, business personnel can more easily obtain the required metering data and conduct in-depth data analysis to provide strong support for business decisions. In addition, through the application sharing, collaboration and release management functions provided by the present invention, it is possible to better promote the sharing and collaboration of data analysis results and further enhance the value of data applications.

[0054] 1) Improved data acquisition efficiency: By opening up the data channel between the energy metering big data service platform and other platforms, real-time access and on-demand extraction of data are achieved, greatly improving the speed and efficiency of data acquisition.

[0055] 2) Improved data quality: With the help of data cleaning and processing tools, abnormal data can be automatically identified and corrected, improving the accuracy and availability of data.

[0056] 3) Enhanced flexibility in data analysis: It provides visual data exploration and correlation analysis tools, supports multi-dimensional sampling and reuse of data analysis paths, and enables analysts to quickly explore data and adjust analysis strategies.

[0057] 4) Strengthened decision-making support capabilities: By building specific application scenarios, such as the prosperity index of small and micro enterprises, the monitoring system of electricity meter manufacturers, and the import and export market analysis report of metering equipment, more accurate and comprehensive data support is provided to decision makers.

[0058] 5) Improved data analysis efficiency: Support for multi-person co-creation and data visualization functions enhances team collaboration efficiency, lowers the threshold for data analysis, and enables non-technical personnel to participate in the data analysis process.

[0059] 6) Maximizing the value of data applications: Through self-service configuration of application scenarios and sharing of analysis results, it promotes the widespread dissemination and utilization of data and maximizes the value of data applications.

[0060] like Figure 3 As shown in the figure, the data processing and analysis tool construction content shows that the data source of the big data platform comes from electricity consumption information collection, MDS, etc., and a general data set is formed through custom data processing, data processing, data management services, etc., and then data sampling, feature engineering, etc. are performed for data analysis and mining; the present invention can adjust the analysis path and multi-dimensional data exploration at any time through path analysis reuse, cross-dimensional jumps, etc., and multiple people can also co-create and share analysis fragments to jointly create a unified analysis environment.

[0061] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0062] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A big data processing and analysis system based on electric energy metering data, characterized in that: include: Data management service module, model service module, self-service analysis module, exploration analysis module and data application management module; The data management service model is used to manage system services, the model service module is used to design algorithm models and manage the designed algorithm models, the self-service analysis module is used to automatically generate SQL based on power energy metering data, the exploration analysis module is used to use the algorithm model designed by the model service module to process the power energy metering data and generate data processing results, and the data application management module is used to manage the power energy metering data.

2. The big data processing and analysis system according to claim 1, characterized in that: The data management service model is used to manage system services, including: Service publishing, service subscription, service authorization and service review.

3. The big data processing and analysis system according to claim 1, characterized in that: The model service module is also used to manage the calling of algorithm models.

4. The big data processing and analysis system according to claim 1, characterized in that: The self-service analysis module automatically generates SQL by graphically dragging, dropping and dropping.

5. The big data processing and analysis system according to claim 1, characterized in that: The exploration and analysis module is specifically used for: Perform data calculation and processing, data mining and model application on electric energy metering data, and display the data in graphical form.

6. The big data processing and analysis system according to claim 1, characterized in that: The data application management module is specifically used for displaying charts and managing drawing boards for electric energy metering data.

7. The big data processing and analysis system according to claim 1, characterized in that: The electric energy metering data includes: General metering data of electric energy.

8. The big data processing and analysis system according to claim 1, characterized in that: The exploration and analysis module is also used to sort out the scope of general support data requirements and build a general data sample market for measurement data based on typical scenario application research requirements; The data sample sets included in the general data sample market of metering data include: metering user archive data sample set, metering equipment archive data sample set, metering equipment full life cycle data sample set, power user daily electricity data sample set and power user load curve data sample set.

9. The big data processing and analysis system according to claim 1, characterized in that: The exploration and analysis module is also used to manage the sample set data, including: Retrieve sample data subsets, view sampling rules, browse sample data, set the accessible scope of specific sample data sets, and take sample data sets offline.

10. The big data processing and analysis system according to claim 1, characterized in that: Also includes: Index release management tool for: Create new analytical indices based on existing indicators, basic data and business rules.

Images