Energy consumption data processing method, system, electronic device and storage medium

Abstract

The present invention provides an energy consumption data processing method, system, electronic device and storage medium, which belongs to the field of big data. The method includes: obtaining energy consumption data that integrates various information systems, including relational data sources, interface data sources, file data sources, big data storage, message queues and non-relational data sources; using the built-in metadata model to collect corresponding metadata in response to the data source selected by the user and the configured collection task, and sorting the metadata to obtain metadata relationships; using a graphical form to provide lineage analysis and full-link analysis of the metadata relationship, wherein the lineage analysis is based on the data flow as the main line, and the full-link analysis characterizes the source, storage, processing process and flow information of the metadata. By integrating and summarizing the data of each heterogeneous system, and by establishing lineage analysis and full-link analysis to achieve data management between different heterogeneous systems and management between different data sources, the data management capability is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an energy consumption data processing method, system, electronic device and storage medium. Background Art

[0002] With the vigorous development of informatization and intelligence, various types and multi-purpose systems play their respective roles in the production and operation activities of high-energy-consuming enterprises such as steel, chemical, industrial parks, ports, and construction industries. However, each system is independent of each other, forming one information island after another. The information between them is not interconnected, and the data coordination between them is low. As a result, users often have to open multiple systems to fully understand the company's production and operation. Or when the data of a heterogeneous system changes, the related data in other systems are not synchronized, and manual repeated data transfer is required.

[0003] In various heterogeneous business systems, the data generated by each heterogeneous system is very large and complex, and the data is constantly superimposed and changed during the operation of the enterprise, resulting in a large amount of duplication of business data, abnormal data volume (such as negative values ​​in normal volume), weak data correlation, etc. At present, there are often hundreds of data tables in heterogeneous systems. During the design process of some data tables, the scenarios of future data mining and analysis are not considered, resulting in a very large amount of data in some single data tables, and complex associations between multiple tables. Once these heterogeneous data are queried, classified, summarized, displayed, etc., the performance will be extremely poor. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide an energy consumption data processing method, system, electronic device and storage medium, which are used to completely or at least partially solve the technical problems existing in the above-mentioned prior art.

[0005] In order to achieve the above object, an embodiment of the present invention provides an energy consumption data processing method, comprising:

[0006] Obtain energy consumption data of various information systems, including relational data sources, interface data sources, file data sources, big data storage, message queues, and non-relational data sources;

[0007] In response to the data source selected by the user and the collection task configured, the corresponding metadata is collected using the built-in metadata model, and the metadata is sorted to obtain metadata relationships;

[0008] The metadata relationship is provided in a graphical form for lineage analysis and full-link analysis, wherein the lineage analysis is based on data flow, and the full-link analysis represents the source, storage, processing and flow information of the metadata.

[0009] Optionally, in response to the data source selected by the user and the configured collection task, collecting corresponding metadata using a built-in metadata model, and arranging the metadata to obtain metadata relationships, includes:

[0010] Analyze the data source type and its structural characteristics selected by the user to ensure that the built-in metadata model covers the characteristics of the data source selected by the user. The metadata model includes but is not limited to database, table model, field, data quality standard, and is used to characterize the attributes and relationships of the data source;

[0011] Configure the parameters of the metadata model collection task according to user needs;

[0012] Establish the relationship between database and table, and between table and field within metadata, where the relationship between database and table, and between table and field are represented as the hierarchical structure and attribute relationship of metadata model;

[0013] If there is a relationship between multiple data sources, a metadata relationship across data sources is established to represent the flow relationship between different data sources.

[0014] Optionally, providing lineage analysis and full-link analysis in a graphical form for the metadata relationship includes:

[0015] Determine the starting point of data lineage analysis based on the data source selected by the user, and track the flow of data from the starting point;

[0016] Extract relevant metadata from various information systems based on data flow;

[0017] With data sources as nodes and data flows as edges, a blood relationship diagram is constructed and displayed in a graphical form;

[0018] Extract relevant metadata related to the entire link from various information systems, where the relevant metadata includes data source information, storage information, processing information, and flow direction information;

[0019] A full-link relationship graph is constructed with data sources, storage systems, and processing systems as nodes, and data flows and processing processes as edges, and the constructed full-link relationship graph is displayed to users in a graphical form.

[0020] On the other hand, the present invention also provides an energy consumption data processing system applied to the energy consumption data processing method, comprising:

[0021] The data integration module is used to connect various information systems and obtain energy consumption data of various information systems. The energy consumption data of various information systems include relational data sources, interface data sources, file data sources, big data storage, message queues and non-relational data sources;

[0022] A data management module, the data management module includes a metadata collection module, a metadata center module, a data map module, a data resource preview module, and a data asset catalog module, wherein the metadata collection module is used to collect corresponding metadata using a built-in collection adapter in response to a user-selected data source and a configured collection task, the metadata center module is used to organize the collected metadata to obtain metadata relationships, the data map module is used to provide lineage analysis and full-link analysis of metadata relationships in a graphical form, the data resource preview module is used to provide asset monitoring in multiple dimensions and present it in a graphical form, and the data asset catalog module is used to support multiple types of energy consumption data source access, data asset inventory, data stratification and domain division, data quality, data lineage, and data service sharing functions.

[0023] Optionally, the data map module is also used to use a metadata distribution map to display data of different types of metadata in different systems, use a system importance statistics map to display the frequency of association between table objects in the system, use a heat statistics map to visualize the usage popularity of metadata and tables, and use a metadata change trend map to display the number of changes of different types of metadata in different time periods.

[0024] Optionally, the data governance module includes a data quality management module, wherein the data quality management module is configured with a rule base for executing quality inspection tasks and saving quality inspection results according to the configured rule base.

[0025] Optionally, the data governance module includes a data security management module, which is configured with a dynamic permission management mechanism for configuring different permissions for different roles, so that users can complete data asset management under the allocation and management of permissions for each role.

[0026] Optionally, the data security management module is also used to perform desensitizing and encryption processing on energy consumption data of different categories and levels according to preset encryption rules to ensure the security of the energy consumption data.

[0027] Optionally, the energy consumption data processing system also includes a data service sharing module for providing an API interface so that users can develop various API services through configurable configuration tools, monitor calls and subscriptions, and for API users, they can self-select the API interface in the service catalog to obtain data.

[0028] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned energy consumption data processing method when executing the program.

[0029] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the energy consumption data processing method described above are implemented.

[0030] Through the above technical solution, the data of various heterogeneous systems are integrated and summarized, and by establishing lineage analysis and full-link analysis, the management of data between different heterogeneous systems and the management between different data sources are realized, thereby improving data management capabilities.

[0031] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0033] Figure 1 is an implementation flow chart of an energy consumption data processing method provided by an embodiment of the present invention;

[0034] Figure 2 is a structural schematic diagram of an energy consumption data processing system provided by an embodiment of the present invention;

[0035] Figure 3 It is a schematic diagram of a rule base interface of an energy consumption data processing system provided by an embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of a data desensitization interface of an energy consumption data processing system provided by an embodiment of the present invention;

[0037] Figure 5 It is a schematic diagram of a data development interface of an energy consumption data processing system provided by an embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of an operation and maintenance interface of an energy consumption data processing system provided by an embodiment of the present invention;

[0039] Figure 7 It is a schematic diagram of the API functional structure of an energy consumption data processing system provided by an embodiment of the present invention;

[0040] Figure 8 It is a software framework diagram of an energy consumption data processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0042] See also Figure 1 As shown, it is a flowchart of an implementation method of an energy consumption data processing method provided by an embodiment of the present invention, including the following execution steps:

[0043] Step 100: Acquire energy consumption data of various information systems, wherein the energy consumption data of various information systems include relational data sources, interface data sources, file data sources, big data storage, message queues, and non-relational data sources;

[0044] Step 101: In response to a data source selected by a user and a collection task configured, corresponding metadata is collected using a built-in metadata model, and the metadata is sorted to obtain metadata relationships.

[0045] Specifically, when executing step 101, the following steps may be specifically performed:

[0046] S1010: Analyze the data source type selected by the user and its structural characteristics, and ensure that the built-in metadata model covers the characteristics of the data source selected by the user, where the metadata model includes but is not limited to database, table model, field, data quality standard, and is used to characterize the attributes and relationships of the data source.

[0047] S1011: According to user requirements, configure the parameters of the metadata model collection task.

[0048] S1012: Establishing the relationship between the database and the table, and the relationship between the table and the field within the metadata, wherein the relationship between the database and the table, and the relationship between the table and the field are represented as the hierarchical structure and attribute relationship of the metadata model.

[0049] S1013: If there is a relationship between multiple data sources, a metadata relationship across data sources is established to represent the flow relationship between different data sources.

[0050] Step 102: Provide lineage analysis and full-link analysis of the metadata relationship in a graphical form, wherein the lineage analysis takes data flow as the main line, and the full-link analysis represents the source, storage, processing and flow information of the metadata.

[0051] Specifically, when executing step 102, the following steps may be specifically performed:

[0052] S1020: Determine the starting point of data lineage analysis based on the data source selected by the user, and track the flow of data from the starting point.

[0053] S1021: Extract relevant metadata from various information systems based on data flow.

[0054] S1022: Construct a blood relationship diagram with data sources as nodes and data flows as edges, and display it in a graphical form.

[0055] S1023: Extract relevant metadata related to the entire link from various information systems, wherein the relevant metadata includes data source information, storage information, processing information, and flow direction information.

[0056] S1024: A full-link relationship graph is constructed with the data source, storage system, and processing system as nodes and the data flow and processing process as edges, and the constructed full-link relationship graph is displayed to the user in a graphical form.

[0057] See also Figure 2 FIG. 1 is a schematic diagram of a structure of an energy consumption data processing system provided by an embodiment of the present invention, including:

[0058] The data integration module 20 is used to connect to various information systems and obtain energy consumption data of various information systems, wherein the energy consumption data of various information systems include relational data sources, interface data sources, file data sources, big data storage, message queues and non-relational data sources.

[0059] Specifically, information systems continue to emerge in the process of enterprise development. ERP, MES, OA, SRM, CRM, financial systems, etc. play an indispensable role in their respective fields. The data of various information systems are connected and integrated for in-depth analysis and application. Information system access mainly includes two methods: interface communication and database docking communication. Common software protocols and types include Restful API, Webservice, MQTT, JMS, kafka, JDBC, ODBC, etc. The system provides self-service ETL tools to support data integration, cleaning and synchronization requirements between heterogeneous information systems. Through the self-service task orchestration method provided, users can configure ETL tasks according to data integration requirements, and integrate the data of heterogeneous information systems into the platform through database connection, standard API, and file acquisition. Provide whole database migration based on CDC technology to solve the problem of massive data synchronization, capture data changes in real time, and synchronize data to the target data source in real time. At the same time, it supports one-time creation of all tables in the source data source and uploads them to the target data source, saving a lot of data synchronization task creation time, and automatically creates tables in the target data source, saving a lot of initialization effort.

[0060] Table 1: Energy consumption data of various information systems

[0061]

[0062] The data management module 21 includes a metadata collection module, a metadata center module, a data map module, a data resource preview module, and a data asset catalog module, wherein the metadata collection module is used to collect corresponding metadata using a built-in collection adapter in response to a data source selected by a user and a configured collection task, the metadata center module is used to organize the collected metadata to obtain metadata relationships, the data map module is used to provide lineage analysis and full-link analysis of metadata relationships in a graphical form, the data resource preview module is used to provide asset monitoring in multiple dimensions and present it in a graphical form, and the data asset catalog module is used to support multiple types of energy consumption data source access, data asset inventory, data stratification and domain division, data quality, data lineage, and data service sharing functions.

[0063] The metadata center module has built-in commonly used metamodels that rarely change, including relational database metamodels, big data storage metamodels, etc., which facilitate the direct creation of metadata collection tasks. It also supports the expansion of business attributes based on the built-in metamodel. In order to improve scalability, the system also supports custom metamodels. Data sources that are not supported by the built-in metamodel can be implemented through custom metamodels. It supports custom multi-level metamodels, such as data source metamodel-table model-field. For metadata that cannot be collected automatically, it supports manual import to enter metadata.

[0064] In some implementations, the metadata management function uniformly merges, organizes, and displays the collected metadata, including business metadata, technical metadata, and management metadata, and supports querying or modifying the basic information, business information, field information, data samples, data lineage, data service sharing, standard quality, and change records of metadata. Covers the entire data flow stage, including database metadata, table metadata, field metadata, indicator metadata, ETL task metadata, data service metadata, etc. Supports filtering according to data source type, searching in the search bar, and publishing metadata to data resources. Centrally displays the collected metadata, supports full-text search, can be viewed according to data type and database, and can dynamically define query conditions. Supports viewing table metadata, such as table name, table type, storage capacity, number of records, belonging library, etc.; also supports maintaining the business metadata of the table, such as business sector, subject domain, business object, etc. Supports viewing field metadata, such as field name, display name, data type, description, etc. Supports viewing data samples. Provides a change record function to record the change process of metadata. Supports viewing data lineage, data service sharing, data quality, and other information. Supports metadata version management, and can view historical versions and compare version differences. Supports custom meta-models and custom business attributes, which facilitates and flexibly expands metadata. Supports metadata change subscription, and can push metadata change information to subscribers.

[0065] Among them, the data map module is also used to use metadata distribution maps to display data of different types of metadata in different systems, use system importance statistics maps to display the association frequency of table objects in the system, use heat statistics maps to visualize the usage popularity of metadata and tables, and use metadata change trend maps to display the number of changes of different types of metadata in different time periods.

[0066] In some embodiments, based on the metadata center, metadata relationships are automatically generated while metadata is automatically collected, and metadata analyses such as lineage analysis, impact analysis, and full-link analysis are provided in a graphical manner. Lineage analysis: Lineage analysis is based on the overall metadata integration of the organization. It is the upstream of metadata analysis and provides metadata analysis across IT systems and BI tools to achieve lineage tracing with data flow as the main line. Impact analysis: Impact analysis can quickly understand the downstream data information of the analysis object, quickly identify the value of metadata, and grasp the possible impact of metadata changes, so as to more effectively assess the risks brought about by the changes, thereby helping users to clean, maintain and use data assets efficiently and accurately. Full-link analysis: Full-link analysis is based on the overall metadata integration of the organization, and comprehensively displays the source, storage and processing process, and flow information of the metadata.

[0067] In some implementations, the overall situation of metadata is monitored, including the overall size, relevance, popularity, changes, etc. of metadata. The metadata distribution graph shows the number of different types of metadata models in different systems. The system importance statistics graph shows the association frequency of table objects in the system, and calculates the frequency of combined use of table-related objects. The heat statistics graph is a visual display of the usage popularity of metadata and the table itself. Every time a user uses / accesses a certain metadata, whether it is a system, table, or field, the system will count it once. The metadata change trend graph shows the number of changes of different types of metadata in different time periods (the past day, the past month, the past year) and different change types (addition, deletion, modification).

[0068] In some implementations, a platform for enterprise data asset management is provided based on metadata, supporting functions such as access to multiple types of data sources, data asset inventory, data stratification and domain division, data quality, data lineage, and data service sharing. As a data asset ledger, it helps enterprises to grasp the status of data assets in a more timely, comprehensive, and in-depth manner. It supports the listing and delisting of data assets, supports the attachment of data assets to database tables, APIs, text files, videos, audios, text files, and report resources and other data types, supports viewing data assets, including basic information, data structure, asset data, data lineage, data service sharing, standard quality, and change records, supports the definition of data directories, supports full-text retrieval, supports the setting of service methods that data assets can provide, such as data API, data push, data query, data download, online analysis, etc., supports the setting of open access scopes for data assets, and can specify personnel and departments.

[0069] In some implementations, indicator definition: Based on business research and data inventory, sort out the indicator system, provide indicator definition capabilities, maintain standard indicator dictionaries, and guide indicator implementation development. Planning documents: Manage planning documents, project planning documents, data standard documents, etc. in the data governance consulting phase.

[0070] In some implementations, the data governance module is also used for data standards management, which are normative constraints that ensure the consistency and accuracy of internal and external use and exchange of data. Data standards management is a series of activities to standardize the formulation and implementation of data standards. It is one of the core activities of data asset management and plays a vital role in improving data quality, clarifying data composition, breaking through data silos, accelerating data circulation, and releasing data value. Data standards management provides capabilities such as data standard entry, import, release, landing mapping, and standard document management, coordinates metadata and data quality, reduces information conflicts and inconsistencies between systems, and improves data quality.

[0071] Including, (1) Standard set management: A standard set is a collection of data standards. Data standards under the same standard set have the same business classification and attributes. Different types of data standards may have different attributes. In order to meet the design of data standards for different projects, the platform provides data standard set management, built-in business attributes, technical attributes, management attributes, quality attributes, life cycle attributes, etc. for users to choose and use, and supports on-demand attribute expansion. (2) Data standard definition: Provide a convenient and flexible operation interface, according to the user's choice of appropriate method, quickly define data standards, standard attributes are consistent with the standard set attributes. Support users to manually create data standards, and support batch creation by importing. By exporting the standard set, users can organize data standards offline. After importing the organized data standards into the platform, they become a mappable and evaluable data standard. (3) Data standard landing: The purpose of the design of data standards is to standardize the data construction of various business systems. The platform supports the setting of landing mapping for data standards. A standard can be mapped multiple times according to actual business needs. The mapping settings are refined to the metadata corresponding to the actual business system, providing a basis for subsequent landing evaluation and data quality monitoring. (4) Data standard documents: You can upload various types of standard documents, including national standards, industry standards, etc., and support multiple formats such as word and excel. Standard documents can be managed uniformly for easy query. When creating data standards, you can associate standard documents as authoritative sources. (5) Standard implementation evaluation: In order to facilitate users to check whether the business system is built in accordance with data standards, the platform provides standard implementation evaluation for data standards. It supports setting scheduled evaluations, and the evaluation execution method supports increments. It supports querying the implementation evaluation details. The mapping of statistical metadata and data standards evaluates the implementation of data standards in the business system and tracks the construction of business system data standards. Built-in implementation evaluation statistics dashbroad analyzes and displays the standard implementation pass rate, and supports filtering according to data standard sets and metadata.

[0072] Data quality is a one-stop platform that supports quality verification, notification and management services for multiple heterogeneous data sources. With data standards as the basis for data verification and metadata as the object of data verification, the process of integrating work links such as quality assessment, quality verification, quality rectification and quality reporting is integrated through simple operation methods such as wizards and visualization to form a complete data quality management closed loop. Specifically, the data governance module includes a data quality management module, wherein the data quality management module is configured with a rule base for executing quality inspection tasks and saving quality inspection results according to the configured rule base.

[0073] In some embodiments, data quality management includes: (1) rule base: data quality rules cover most scenarios of data quality issues, see Figure 3As shown in the figure, it includes rules such as null value check, duplicate value check, value range check, data format check, enumeration value check, SQL search, cross-source search, and consistency search. It also supports the creation of rule tasks based on data standards, automatically matching the attribute content of data standards with corresponding rules, and forming a comprehensive monitoring management. (2) Quality inspection tasks: Data quality inspection tasks support the definition of inspection rules, inspection time, problem weights, scoring rules, etc. At the same time, tasks support manual execution, scheduled execution, and ETL task trigger execution. (3) Execution records: Through quality monitoring, quality results will be generated and saved, including: monitoring object name, execution time, number of execution rules, number of abnormal rules, results, etc. The system will automatically generate a detailed result table for each quality inspection plan, and calculate the data quality score based on the scoring rules, weights, problem levels, number of error records, and other variable factors. The results of the inspection can be alerted through internal messages, emails, SMS (requires SMS equipment), etc., and the problem handling process can be automatically or manually initiated. (4) Quality analysis: After the quality inspection task is completed, the results are integrated and output, and the content presented includes the distribution of data quality issues, the comprehensive data quality score, and the top 10 rules for data quality issues.

[0074] In some embodiments, data security management is to plan, formulate, and implement relevant security policies and procedures to ensure that data and information assets have appropriate authentication, authorization, access, and auditing measures during use. Effective data security policies and procedures ensure that the right people use and update data in the right way, and restrict all inappropriate access and update of data. The system supports data permission management for users based on actual business needs, and the system supports data desensitization and encryption based on security rules. The data governance module also includes a data security management module, which is used to classify the energy consumption data of various information systems according to business dimensions and to perform graded processing according to the sensitivity of energy consumption data, and obtain energy consumption data of different categories and levels, wherein the business dimension includes but is not limited to industry dimension, business field dimension, data source dimension, sharing dimension, and data openness dimension.

[0075] Specifically, data classification and grading is a prerequisite for data security management and provides support for data security management.

[0076] Data classification is to group data with the same attributes or characteristics together to form different categories, so that people can query, identify, manage, protect and use data through categories. Data classification is more from the perspective of business or data management, such as: industry dimension, business field dimension, data source dimension, sharing dimension, data openness dimension, etc. According to these dimensions, data with the same attributes or characteristics are classified according to certain principles and methods. Data classification is defined according to certain principles and methods based on the sensitivity of the data and the impact on the victim after the data is tampered with, destroyed, leaked or illegally used. Data classification is more from the perspective of security compliance requirements and data protection requirements. Data classification is essentially data classification in the data sensitivity dimension.

[0077] In some implementations, in the authority management of role functions and asset analysis results, after logging in to user accounts under different roles according to the user's authority management for the role, the data assets and related data results seen are different, and the operating permissions are also different. It is ensured that users can easily and efficiently complete the allocation and management of asset content under the allocation and management of permissions for each role, and the authority division of multiple roles such as owners, administrators, member roles and external personnel is achieved safely and effectively. The data security management module is configured with a dynamic authority control mechanism for configuring different permissions for different roles, so that users can complete the management of data assets under the allocation and management of permissions for each role.

[0078] In some implementations, in addition to managing user access rights to data, the system also authorizes and controls access to data callers (such as APPs and third-party business systems), which can effectively ensure the stability and security of platform data services. The authorization page provides two authorization methods: whitelist and blacklist. It supports two modes: ID authorization and IP authorization. It supports secondary editing and disabling of authorized services. It also provides advanced configuration of authorization, and provides settings for key parameters such as flow control and number limit.

[0079] In some embodiments, the data security management module is further used to perform desensitizing and encryption processing on energy consumption data of different categories and levels according to preset encryption rules to ensure the security of the energy consumption data.

[0080] For details, see Figure 4As shown, data desensitization and encryption supports users to customize data security rules according to actual security needs. The system has preset mainstream advanced data security encryption rules, including data obfuscation, data padding, data encryption and other rules; among them, data obfuscation desensitization can desensitize and deform sensitive information, and can be quickly implemented without changing the business process of the enterprise, ensuring the validity, integrity and relevance of the desensitized data, so as to improve the security of data assets in data storage, data flow and other links.

[0081] In some embodiments, the energy consumption data processing system also includes a data warehouse development module. First, data modeling is performed. In the process of information construction, the data of an enterprise increases exponentially. The data volume is huge and complex, and the standards of various types of data are inconsistent, which often makes data difficult to manage. The data modeling service of the data governance platform defines and manages disordered, messy, cumbersome, huge and difficult to manage data in a structured and standardized manner. By building an enterprise data warehouse, data assets are formed to support the maximization of data value. Specifically, (1) for the global data of the enterprise, data warehouse planning capabilities are provided to plan business segments, subject domains, and data stratification to achieve layered and domain-based management of data assets. (2) for relational data and object-oriented data in the data lake, data warehouse construction capabilities are provided to support the design and creation of dimension logic tables, fact logic tables, and summary logic. (3) Reverse creation is supported to batch create data models using the data structure of existing physical tables. (4) "Design is development" is realized, and ETL tasks are automatically generated according to the materialized configuration of the model without writing a line of code, which greatly reduces the construction threshold of the data warehouse. (5) Traditional modeling tools are mainly oriented towards design, while the data modeling of the data governance platform innovatively integrates the concept of data governance, promotes data governance into the development process, and conducts source governance of the development state, so that business personnel can easily participate in the definition of data standards and formulate business rules. It solves the problem of standard implementation and fundamentally controls the quality of incremental data in the enterprise. (6) It supports the construction of real-time data warehouses, based on Flink as the computing engine, to perceive data changes in the ODS layer in real time and update streaming data to the upper-level model. (7) As a key link in the data link, the data model shields complex technical details downward and connects with functional modules such as data asset catalog, indicator management, data quality, and data API upward, effectively supporting data governance scenarios and data application scenarios. (8) It provides version management and supports version comparison to facilitate tracing the change process of the model.

[0082] Secondly, carry out data development: first, conduct a task overview, provide multi-dimensional task monitoring, and count the task scale according to dimensions such as task development, task operation and maintenance, task implementation overview, task execution result distribution, task execution status and trend, and task exception tracking. And present it in an intuitive chart to facilitate managers to control the operation of tasks. Secondly, establish task lineage. Task lineage is an analysis of the origin of table metadata and its dependencies. It reflects the source and processing of the data, and describes what happens to the data in different processes. Through lineage analysis, you can understand the upstream data source that the current table metadata depends on, the ETL task that outputs the data, and the data flow path, thereby assisting data openers in locating data and locating problems. Then carry out task development. For its data development interface, see Figure 5 As shown in the figure, it supports self-service creation of ETL tasks. It adopts a configurable approach, so users do not need to write a line of code and can flexibly arrange tasks to meet the differentiated scenarios and needs of data synchronization, cleaning, and processing in different project sites. X-ETL provides a variety of operator components. Each operator is a data processing unit, and multiple operators are combined into ETL tasks. Currently, it supports 29 operators in Table 2, which can cover most data processing scenarios. Finally, perform task operation and maintenance. For its operation and maintenance interface, see Figure 6 As shown, it supports quick release and adjustment release modes. The quick release mode can support the release and operation of successfully debugged tasks. The adjustment release mode allows the modification of data input and output sources to achieve the switch from the test process to the actual production operation. All released tasks are displayed in the release task list. Users can view the relevant information and operation status of the tasks through the release task list. It supports the operation status and process monitoring of the release tasks, provides task execution history logs, task execution results, task exception reasons, and exception data, and supports manual triggering and re-execution of abnormal tasks. It supports viewing task dependencies, jumping to task lineage, and displaying the upstream and downstream dependencies of tasks.

[0083] Table 2: 29 operators

[0084]

[0085] In some embodiments, the energy consumption data processing system also includes an indicator management module, which serves as a comprehensive management center for data indicators and carries the business needs, technical needs, and management needs of indicators. Through standardized definition and standardized development of indicators, an enterprise data indicator system is built, and indicator data results are implemented. At the same time, applications such as indicator query and services are provided to eliminate data ambiguity, reduce the communication cost between business and technology, and make indicator data visible, usable, and manageable. Through indicator management, users can develop indicators, calculate indicators, and publish indicators online without code, obtain all current indicator assets, and develop indicators that do not exist, such as personnel performance indicators, production indicators, quality indicators, inventory indicators, sales indicators, and energy consumption indicators for various business indicators. (1) Indicator development. The data source for indicator development is mainly the data warehouse, which provides data storage and computing capabilities for indicator management. There are two main ways to develop indicators: codeless development of indicators. Based on the data model of the data warehouse, indicators are configured and generated in the form of page visualization operations, which greatly reduces the threshold for indicator development and can create most indicator types. Indicators can be flexibly created by writing SQL, supporting complex SQL query statements, integrating data from several database tables into one view, facilitating the picking of indicators containing specific functions such as sum, count, etc. Multiple indicators can be established at one time, making the creation of indicators more flexible and efficient. (2) Indicator data output: In addition to supporting online query of indicator data, indicator management also supports data output to the target data source to meet the needs of indicator result display, analysis and sharing. Currently, it supports publishing indicator data to data services. When publishing to data services, the indicator API is generated online and the corresponding call address is automatically generated. It is convenient for third parties to call the API interface to obtain the corresponding indicator data information.

[0086] In some embodiments, the energy consumption data processing system further includes a data service sharing module for providing an API interface. Figure 7 As shown, users can develop various API services through configuration tools, monitor calls and subscriptions, and API users can select API interfaces in the service catalog to obtain data.

[0087] Specifically, API is first generated through MySQL, SQL server, Oracle, object model, time series database and graph database, followed by later API configuration including template wizard mode, custom SQL mode and API testing, multi-type API registration and API testing, and then the API is published to the API gateway for API calls across regions, across enterprises or within the enterprise. It also includes API management, including authorization approval, API details viewing and API call viewing, and API security including call flow control and security encryption.

[0088] In some implementations, the environmental deployment requirements of the energy consumption data processing system are preferably configured as 8 servers, of which 2 energy consumption big data management software, 4 data warehouses, and 2 data integration exchange tools are deployed. The specific configuration and operating system and other specification requirements are shown in Table 3:

[0089] Table 3: Configuration and operating system specifications

[0090]

[0091] See also Figure 8 As shown in FIG. 1 , a software framework diagram of an energy consumption data processing system provided by an embodiment of the present invention is mainly divided into the following layers from bottom to top:

[0092] Data source layer: mainly used to manage the connection information of various data sources, such as third-party systems, interface APIs, hardware equipment, OA, ERP, MES, LIMS, WMS, data warehouses and other business system data, time series data, enterprise data center data, etc.

[0093] Big data storage and computing: The storage and computing base of the big data platform provides integrated lake and warehouse storage support, which can uniformly store data scattered in various systems in the enterprise, with flexible deployment, elastic expansion, and low operation and maintenance costs. It supports real-time data warehouses, based on MPP databases, can update data in real time and query at high speed, and supports OLAP multi-dimensional analysis and high-concurrency queries.

[0094] Data integration and exchange: Supports access to / from multi-source heterogeneous data, using a low-code free configuration mode. Quickly upload all tables in the source data source to the data warehouse to achieve whole-database migration, saving a lot of time in creating real-time data synchronization tasks, supporting real-time collection and offline incremental synchronization, and also performing data preprocessing and data exchange.

[0095] Deep data processing (data factory): including data warehouse design: providing a series of low-code data development tools such as data visualization modeling, data development and indicator development, supporting low-threshold and standardized construction of data warehouses, subject table management, model management and table physicalization; indicator design: model construction of indicators, SQL construction of indicators, indicator query and indicator API.

[0096] Data asset management and data governance: Provides metadata, data lineage, data asset catalog, data standards, data quality, data security, standard models, data lifecycle management and other functions to support the implementation of scenarios such as data asset management and data governance (data resource statistics, asset catalog, asset full-text retrieval, asset services, application and authorization).

[0097] Data service layer: helps enterprises open data assets to the outside world safely and efficiently, provides the ability to quickly generate APIs from data tables, and supports the rapid registration of existing APIs to the data service platform for unified data query, access authentication, and traffic control management and release.

[0098] Data application layer: This layer mainly uses the platform's convenient application functions, such as large-screen visualization, self-service analysis, object tools, and drag-and-drop multi-dimensional data acquisition, to better lower the threshold for data use, explore data value, and support decision-making.

[0099] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned energy consumption data processing method when executing the program.

[0100] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the energy consumption data processing method described above are implemented.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0108] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0109] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for processing energy consumption data, characterized in that: include: Obtain energy consumption data of various information systems, including relational data sources, interface data sources, file data sources, big data storage, message queues, and non-relational data sources; In response to the data source selected by the user and the collection task configured, the corresponding metadata is collected using the built-in metadata model, and the metadata is sorted to obtain metadata relationships; The metadata relationship is provided in a graphical form for lineage analysis and full-link analysis, wherein the lineage analysis is based on the data flow, and the full-link analysis represents the source, storage, processing and flow information of the metadata; Among them, data standards are used as the basis for data verification, and metadata is used as the object of data verification. Through wizard-based, visual and simple operation methods, the quality assessment, quality verification, quality rectification and quality reporting work links are integrated into the process to form a complete data quality management closed loop. Data quality management includes: Rule library: Data quality rules cover scenarios of data quality problems, including null value check, duplicate value check, value range check, data format check, enumeration value check, SQL search, cross-source search and consistency search rules. At the same time, it supports the creation of rule tasks based on data standards, automatically matching the attribute content of data standards with corresponding rules, and forming comprehensive monitoring management; Quality inspection tasks: Data quality inspection tasks support the definition of inspection rules, inspection time, and problems. Weights and scoring rules; at the same time, tasks support manual execution, scheduled execution, and ETL task trigger execution; Execution records: Through quality monitoring, quality results will be generated and saved, including: monitoring object name, execution time, number of execution rules, number of abnormal rules, and results. The system automatically generates a detailed result table for each quality inspection plan, and calculates the data quality score based on the scoring rules, weight, problem level, and variable factors of the number of error records; the inspection results are alerted through station letters, emails, and SMS, and the problem handling process is automatically or manually initiated; Quality analysis: After the quality inspection task is completed, the results are integrated and output, and the presentation content includes data quality problem distribution, data quality comprehensive score, and data quality problem TOP10 rules; Among them, the metadata distribution map is used to display the data of different types of metadata in different systems, the system importance statistics map is used to display the association frequency of table objects in the system, the heat statistics map is used to visualize the use heat of metadata and tables, and the metadata change trend map is used to display the number of changes of different types of metadata in different time periods. The step of collecting corresponding metadata using a built-in metadata model in response to the data source selected by the user and the configured collection task, and arranging the metadata to obtain metadata relationships includes: Analyze the data source type and its structural characteristics selected by the user to ensure that the built-in metadata model covers the characteristics of the data source selected by the user. The metadata model includes but is not limited to database, table model, field, data quality standard, and is used to characterize the attributes and relationships of the data source; Configure the parameters of the metadata model collection task according to user needs; Establish the relationship between database and table, and between table and field within metadata, where the relationship between database and table, and between table and field are represented as the hierarchical structure and attribute relationship of metadata model; If there is a relationship between multiple data sources, a metadata relationship across data sources is established to represent the flow relationship between different data sources.

2. The energy consumption data processing method according to claim 1, characterized in that: The providing blood relationship analysis and full link analysis of the metadata relationship in a graphical form includes: Determine the starting point of data lineage analysis based on the data source selected by the user, and track the flow of data from the starting point; Extract relevant metadata from various information systems based on data flow; With data sources as nodes and data flows as edges, a blood relationship diagram is constructed and displayed in a graphical form; Extract relevant metadata related to the entire link from various information systems, where the relevant metadata includes data source information, storage information, processing information, and flow direction information; A full-link relationship graph is constructed with data sources, storage systems, and processing systems as nodes, and data flows and processing processes as edges, and the constructed full-link relationship graph is displayed to users in a graphical form.

3. An energy consumption data processing system applied to the energy consumption data processing method according to any one of claims 1 to 2, characterized in that: include: The data integration module is used to connect various information systems and obtain energy consumption data of various information systems. The energy consumption data of various information systems include relational data sources, interface data sources, file data sources, big data storage, message queues and non-relational data sources; A data management module, the data management module includes a metadata collection module, a metadata center module, a data map module, a data resource preview module, and a data asset catalog module, wherein the metadata collection module is used to collect corresponding metadata using a built-in collection adapter in response to a user-selected data source and a configured collection task, the metadata center module is used to organize the collected metadata to obtain metadata relationships, the data map module is used to provide lineage analysis and full-link analysis of metadata relationships in a graphical form, the data resource preview module is used to provide asset monitoring in multiple dimensions and present it in a graphical form, and the data asset catalog module is used to support multiple types of energy consumption data source access, data asset inventory, data stratification and domain division, data quality, data lineage, and data service sharing functions.

4. The energy consumption data processing system according to claim 3, characterized in that: The data map module is also used to use a metadata distribution map to display the data of different types of metadata in different systems, use a system importance statistics map to display the association frequency of table objects in the system, use a heat statistics map to visualize the usage popularity of metadata and tables, and use a metadata change trend map to display the number of changes of different types of metadata in different time periods.

5. The energy consumption data processing system according to claim 3, characterized in that: The energy consumption data processing system also includes a data governance module, which includes a data quality management module, wherein the data quality management module is configured with a rule base for executing quality inspection tasks and saving quality inspection results according to the configured rule base.

6. The energy consumption data processing system according to claim 3, characterized in that: The energy consumption data processing system also includes a data security management module, wherein the data security management module is configured with a dynamic permission management mechanism for configuring different permissions for different roles, so that users can complete the management of data assets under the allocation management of permissions for each role.

7. The energy consumption data processing system according to claim 6, characterized in that: The data security management module is also used to perform desensitizing and encryption processing on energy consumption data of different categories and levels according to preset encryption rules to ensure the security of the energy consumption data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the energy consumption data processing method as described in any one of claims 1-2 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the energy consumption data processing method according to any one of claims 1 to 2 are implemented.

Images