Integrated system, data processing method and equipment for power dispatch domain, and storage medium

Abstract

The application relates to a power dispatch domain-oriented integrated system, a data processing method, equipment and a storage medium. The system comprises a physical isolation device arranged between a safety area 2 and a safety area 3 for one-way data transmission, a data exchange management module arranged in the safety area 2, and a quality management module, a resource management module, an asset management module and a value service module arranged in the safety area 3; the data exchange management module acquires multi-source heterogeneous dispatch data and one-way transits to the safety area 3 through the physical isolation device, sequentially performs quality checking, hierarchical storage, and realizes resource-to-asset conversion through a resource directory and metadata, and provides standardized data services to an upper-layer application, the five modules are sequentially connected, and a data processing link integrated with access, management, storage, assets and services is formed. The scheme can uniformly access and dispatch multi-source heterogeneous data, realizes safe cross-area transmission, and effectively solves the problem that dispatch data is difficult to be uniformly fused and managed.

Description

Technical Field

[0001] This application relates to the field of power big data technology, and in particular to an integrated system, data processing method, equipment and storage medium for the power dispatch domain. Background Technology

[0002] Driven by the big data strategy and the construction of digital power grids, the power grid dispatch domain has accumulated massive amounts of heterogeneous data from multiple sources, including parameter models, real-time operation data, equipment ledgers, metering data, meteorological data, and market operations data. This data forms the core foundation for safe power grid operation, load forecasting, electricity spot trading, and auxiliary decision-making. However, existing systems in the dispatch domain (such as SCADA, OCS, and OMS, where SCADA is the data acquisition and monitoring control system, OCS is the dispatch operation control system, and OMS is the power outage management system) are independent of each other, with data access, storage, and sharing all adopting a decentralized model, and a unified data processing platform has not yet been formed. At the same time, the power monitoring system is divided into security zone 2 and security zone 3 according to security protection specifications, and only one-way data transmission is allowed between the two zones. Existing general data platforms are not adapted to this partitioning and isolation constraint. Constrained by the decentralized nature of existing systems and the physical isolation across security zones, current technologies cannot integrate the fusion, quality control, and asset services of multi-source data into an integrated data processing link, making it difficult for dispatch data to be uniformly accessed, effectively managed, and transformed into data assets that can be shared externally. Summary of the Invention

[0003] Based on this, it is necessary to provide an integrated system, multi-source data processing method, computer equipment, computer-readable storage medium, and computer program product that can connect access, governance, storage, assets, and services, and is adapted to the cross-regional physical isolation between the second and third power security zones, and is oriented towards the power dispatch domain, to address the above-mentioned technical problems.

[0004] Firstly, this application provides an integrated system for the power dispatch domain, comprising:

[0005] The system includes a physical isolation device, a security zone 2 side module, and a security zone 3 side module. The physical isolation device is deployed between the power monitoring security zone 2 and security zone 3 and is used to realize unidirectional data transmission. The security zone 2 side module is deployed in the security zone 2, and the security zone 3 side module is deployed in the security zone 3.

[0006] The second security zone module includes a data exchange management module, which is used to acquire scheduling data from multi-source heterogeneous data sources in the power dispatch domain, parse and preprocess the scheduling data, and transfer the processed scheduling data unidirectionally to the third security zone through the physical isolation device.

[0007] The three-zone security module includes: a data quality management module for quality verification of the scheduling data; a data resource management module for hierarchical storage of the quality-verified scheduling data according to processing depth and business attributes; a data asset management module for constructing resource catalogs and metadata for the hierarchically stored scheduling data; and a data value service module for providing standardized data services to upper-layer applications in the power dispatch domain based on the resource catalogs and metadata constructed by the data asset management module.

[0008] The data exchange management module, data quality management module, data resource management module, data asset management module, and data value service module are connected in sequence to form an integrated system of access, governance, storage, assets, and services.

[0009] In one embodiment, the data exchange management module includes at least one of a data source management unit, a policy configuration management unit, a task scheduling management unit, and a data acquisition unit:

[0010] The data source management unit is used to uniformly configure and monitor the status of multi-source heterogeneous data sources, including interface classes, file classes, database classes, and message middleware classes.

[0011] The strategy configuration management unit is used to configure at least one of the following collection modes: full, incremental, real-time, and batch, as well as data extraction, format conversion, field filtering, and data routing rules.

[0012] The data acquisition unit is used to parse, reconcile, and clean the scheduling data under the drive of the data source management unit and the policy configuration management unit.

[0013] The task scheduling management unit is used to perform task slicing, multi-threaded parallel transmission, and breakpoint resumption scheduling control on the scheduled data to be transmitted, providing distributed scheduling capabilities for the one-way file transfer of the physical isolation device.

[0014] In one embodiment, the process by which the data quality management module performs quality verification on the scheduling data includes:

[0015] Based on the scheduling-specific data quality rule library, the scheduling data is verified according to three levels of verification dimensions: field level, record level, and inter-table level.

[0016] Among them, field-level validation verifies the format, value range, length, and required fields of a field; record-level validation verifies the total number of rows in a single table, data uniqueness, and logical relationships between fields; and inter-table-level validation verifies primary and foreign key relationships, bidirectional data matching, and cross-table logical consistency between multiple data tables.

[0017] The data quality management module is also used to generate rectification work orders when verification anomalies occur and push them to the corresponding business responsible persons, receive the issuance, processing, feedback, withdrawal or removal operations performed by the business responsible persons on the rectification work orders, and perform statistical analysis on the number and processing rate of rectification work orders according to business and time dimensions.

[0018] In response to the problems of fragmented existing data quality control systems, which only support single-entry verification and lack abnormal alarms and work order closure processes, this embodiment divides the verification into three levels: field level, record level, and table level. It also incorporates a dedicated scheduling rule base and a full-process work order closure to make data problems traceable and rectification closed-loop.

[0019] In one embodiment, the data resource management module is specifically used to store the scheduling data into a five-level hierarchical storage architecture consisting of a raw database, a resource database, a topic database, a business database, and a model database, including:

[0020] The original library is used to store the accessed scheduling data as is and to serve as the basis for data traceability.

[0021] The resource library is used to clean, standardize, and integrate the scheduling data in the original library to form a shared basic data across the entire domain.

[0022] The subject library is used to integrate the dispatch data in a targeted manner around the core business themes of power dispatch;

[0023] The business library is used to store exclusive data for specific scheduling business scenarios and to work with the tag library for multi-dimensional tagging management.

[0024] The model library is used to store power dispatching professional data models according to the four-step modeling process of business modeling, conceptual modeling, logical modeling and physical modeling.

[0025] In response to the problems of the existing system's chaotic and hierarchical data storage architecture, which makes it impossible to achieve data traceability and classification reuse, this embodiment implements five-level hierarchical persistence based on the depth of data processing and business attributes, taking into account traceability, reuse and business adaptation.

[0026] In one embodiment, the data asset management module further includes at least one of a data resource catalog subunit, a metadata management subunit, a data standard management subunit, and a data operation and maintenance subunit;

[0027] The data asset management module does not physically store the scheduling data. Instead, it uses the scheduling data stored hierarchically by the data resource management module as the control object. The scheduling data is cataloged, reviewed, published, and subscribed to through the data resource catalog subunit. Technical metadata and business metadata are collected and managed through the metadata management subunit. A three-way mapping between standards, metadata, and quality rules is established through the data standard management subunit. On the storage platform of the data resource management module, the scheduling data is transformed into a controllable, shareable, and traceable data asset.

[0028] To address the issue of incomplete data assetization in most power data systems due to a lack of data storage capabilities and insufficient metadata, data standards, resource catalogs, and data operation and maintenance (O&M) capabilities, this embodiment addresses this problem by cataloging, describing, and governing data at the storage layer through four sub-units: resource catalogs, metadata, data standards, and data O&M. It's important to note that the data resource management module and the data asset management module form a collaborative upper-lower layer relationship between storage foundation and asset management: the data resource management module is responsible for scheduling the physical layered storage of data, resolving the questions of where and how data is stored, thus providing a data resource foundation; the data asset management module itself does not physically store business data, but rather uses the scheduling data stored layered by the data resource management module as the object of management. Through resource catalogs, metadata, data standards, and data O&M, it catalogs, describes, standardizes, and maintains these resources, resolving the issues of how data is cataloged, understood, and governed, thereby transforming data residing at the resource layer into manageable, shareable, and traceable assets.

[0029] In one embodiment, the data value service module includes a directory service subunit, a call service subunit, and an intelligent retrieval service subunit, comprising:

[0030] The directory service subunit is used for cataloging, reviewing, publishing, and subscribing to data resources, and for achieving secure data sharing based on hierarchical permissions. The call service subunit is built on a distributed data service bus based on the REST application programming interface, providing data interaction covering messages, data streams, files, and audio and video, and has the capabilities of service routing, permission control, call rate limiting, and call log auditing. The intelligent retrieval service subunit integrates natural language processing and knowledge graph technologies to perform unified semantic retrieval of structured and unstructured data.

[0031] In response to the problems of existing platforms having limited data service capabilities and only providing basic queries, making it difficult to unlock the value of data elements, as well as the problems of poor power security adaptability of general platforms and failure to meet the standards for transmission latency and protocol adaptation across physical isolation devices, this embodiment constructs an integrated service system of directory subscription, interface call and intelligent semantic retrieval, and realizes one-way, low-latency cross-zone service data interaction based on the physical isolation device between security zone 2 and security zone 3.

[0032] Secondly, this application also provides a multi-source data processing method, including:

[0033] Acquire scheduling data from multiple heterogeneous data sources in the power dispatch domain;

[0034] The acquired scheduling data is parsed and preprocessed in the data exchange management module deployed in Security Zone 2;

[0035] Through the physical isolation device between Security Zone 2 and Security Zone 3, the processed scheduling data is transmitted to the data quality management module, data resource management module, data asset management module and data value service module deployed in Security Zone 3 via a one-way file transfer method;

[0036] The scheduling data is quality verified based on the data quality management module.

[0037] Based on the data resource management module, the quality-verified scheduling data is stored in layers according to processing depth and business attributes;

[0038] Based on the data asset management module, a resource catalog and metadata are constructed from the scheduling data stored hierarchically by the data resource management module, thereby realizing the transformation of scheduling data from resources to assets;

[0039] The data value service module provides standardized data services to upper-layer applications in the power dispatch domain. These steps are sequentially linked, forming an integrated data processing chain for multi-source heterogeneous data sources within the power dispatch domain.

[0040] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned method.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned method.

[0042] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned method.

[0043] The aforementioned integrated system, method, computer equipment, computer-readable storage medium, and computer program products for the power dispatch domain, through one-way data exchange, quality governance, hierarchical storage, assetization, and data services in a coordinated manner between security zones two and three, integrate the fusion, quality control, and asset services of multi-source data in the power dispatch domain into an integrated data processing link. This enables unified access to multi-source heterogeneous dispatch data and secure cross-zone transmission, thereby solving the problem that dispatch data in the power dispatch domain is difficult to unify, effectively govern, and transform into externally serviceable assets, and improving the accuracy, consistency, and reusability of dispatch data. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a diagram illustrating the application environment of an integrated platform in one embodiment.

[0046] Figure 2 This is a schematic diagram of the architecture of an integrated platform in one embodiment;

[0047] Figure 3 This is a flowchart illustrating a multi-source data processing method in one embodiment;

[0048] Figure 4 This is a flowchart of distributed scheduling and breakpoint resumption during cross-regional transmission of the data exchange management module in one embodiment;

[0049] Figure 5 This is a flowchart of the three-layer verification and work order closure process of the data quality management module in one embodiment;

[0050] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0053] It should be noted that the meanings of the technical terms used in this application are as follows:

[0054] Security Zone 2, also known as the non-control zone for power monitoring system production, is where non-control-related services such as data acquisition and exchange are deployed. Security Zone 3, also known as the production management zone, is where data governance and data services for business and maintenance personnel are deployed. Only one-way data transmission is allowed between the two zones, and cross-zone isolation is achieved through forward and reverse physical isolation devices.

[0055] Physical isolation devices refer to isolation equipment deployed between security zone 2 and security zone 3, which only allows data to flow in one direction.

[0056] SFTP, or SSH File Transfer Protocol, is used for transferring standard data files across regions.

[0057] Metadata refers to data that describes data, including technical metadata and business metadata; data assets refer to cataloged, standardized, and governed scheduling data that can be shared and served.

[0058] REST API, or Representational StateTransfer Application Programming Interface, is used to build a distributed data service bus.

[0059] NLP stands for Natural Language Processing; knowledge graphs refer to semantic networks that describe entities and their relationships using a graph structure, used for unified semantic retrieval.

[0060] The integrated platform (also known as an integrated system) for the power dispatch domain provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown includes, in sequence, an external data source, a scheduling data network / integrated data network, a second security zone, a physical isolation device, a third security zone, and an internal network terminal user / scheduling domain upper-layer application.

[0061] The external data sources of the dispatch domain (including SCADA, OCS, OMS, meteorological systems, etc., where SCADA is the data acquisition and monitoring control system, OCS is the dispatch operation control system, and OMS is the power outage management system) communicate with the data exchange management module deployed in the second safety zone via the dispatch data network or integrated data network. The data exchange management module, through physical isolation devices deployed between the second and third safety zones, unidirectionally transfers the processed dispatch data to the third safety zone-side module, which further provides standardized data services to the upper-level applications of the dispatch domain. The physical isolation devices include forward and reverse physical isolation devices to achieve unidirectional data transmission across safety zones. The upper-level applications of the dispatch domain include dispatch business systems such as load forecasting, electricity spot market, situational awareness, and equipment monitoring. The data storage system can store the data processed by the data exchange management module and the third safety zone-side module. The data storage system can be integrated on the corresponding server or placed in a distributed storage cluster or other network server.

[0062] In one exemplary embodiment, Figure 2 The diagram shows the architecture of an integrated system for the power dispatch domain. As shown, the integrated system includes: a security zone 2 side module, a physical isolation device, and a security zone 3 side module. The physical isolation device is deployed between the power monitoring security zone 2 and security zone 3 and is used to realize unidirectional data transmission. The security zone 2 side module is deployed in security zone 2, and the security zone 3 side module is deployed in security zone 3.

[0063] The second security zone module includes a data exchange management module, which is used to acquire scheduling data from multiple heterogeneous data sources, parse and preprocess the scheduling data, and transfer the processed scheduling data to the third security zone in one direction through a physical isolation device.

[0064] The three-zone security module includes: a data quality management module, used to perform quality verification on dispatch data and generate rectification work orders to drive closed-loop rectification of abnormal data when verification is abnormal; a data resource management module, used to store the quality-verified dispatch data in layers according to processing depth and business attributes; a data asset management module, used to build resource catalogs and metadata for the dispatch data stored in layers by the data resource management module, realizing the transformation of dispatch data from resources to assets; and a data value service module, used to provide standardized data services to upper-layer applications in the power dispatch domain based on assetized data.

[0065] The aforementioned data exchange management module, data quality management module, data resource management module, data asset management module, and data value service module can be connected or cooperated in sequence to form an integrated data processing link of multi-source data access—governance—storage—asset—service.

[0066] Optionally, the data exchange management module also includes a data source management unit, a strategy configuration management unit, a task scheduling management unit, and a data acquisition unit. The data source management unit performs unified configuration and status monitoring for data sources of interface type, file type, database type, and message middleware type. The strategy configuration management unit configures full, incremental, real-time, and batch acquisition modes, as well as data extraction, format conversion, field filtering, and data routing rules. The data acquisition unit parses, reconciles, and cleans the scheduled data.

[0067] The data source management unit, policy configuration management unit, data acquisition unit, and task scheduling management unit work together in the order of data source configuration—acquisition rules—acquisition execution—distributed scheduling transmission: The data source management unit provides unified data source configuration and status monitoring for acquisition; the policy configuration management unit configures full, incremental, real-time, and batch modes for acquisition, as well as extraction, transformation, filtering, and routing rules; the data acquisition unit, driven by the data source management unit and the policy configuration management unit, parses, reconciles, and cleans the scheduled data; the task scheduling management unit, based on a distributed architecture consisting of a scheduling center and an executor cluster, performs task slicing, multi-threaded parallel transmission, and breakpoint resumption scheduling control on the scheduled data to be transmitted, providing distributed scheduling capabilities for unidirectional file transfer of physically isolated devices.

[0068] In one exemplary embodiment, such as Figure 4 As shown, the process by which the data exchange management module unidirectionally transfers the processed scheduling data to the third security zone through the physical isolation device includes: employing a distributed architecture consisting of a scheduling center and an executor cluster to slice the scheduling data to be transmitted and execute data transmission in a multi-threaded parallel manner; monitoring server resources and data traffic during transmission, recording the breakpoint position when the network or storage is interrupted, and resuming transmission from the recorded breakpoint position after the network or storage is restored to avoid full retransmission; and using a file transfer protocol through the physical isolation device to transfer the scheduling data from the second security zone to the third security zone in a unidirectional file transfer manner.

[0069] The aforementioned task slicing, along with the multi-threaded parallel execution of data transmission and monitoring of server resources and data traffic, and the resumption of transmission after interruption, together constitute the distributed transmission execution process within the secure zone 2. The resumption of transmission after interruption is used to resume only the remaining slices after network or storage interruption to avoid full retransmission. These two steps provide a complete and reliable data source for one-way file transfer through physical isolation devices. The three steps are connected in the order of slice parallel transmission, interruption resumption, and isolation transfer to jointly achieve reliable cross-zone data transmission under physical isolation constraints.

[0070] In response to the problems of insufficient multi-source data access and scheduling capabilities of existing scheduling systems, and the need for full retransmission after network interruption leading to low transmission efficiency and high risk of data loss, this embodiment reduces the transmission loss rate to zero through distributed task slicing, multi-threaded parallel transmission, and breakpoint resumption. Standard data files are generated in the second security zone and transferred to the third security zone via the SFTP protocol and physical isolation device. The third security zone automatically parses and stores the data, meeting the power cross-regional transmission latency requirements (forward isolation less than 3 seconds, reverse isolation less than 20 seconds, for non-large file scenarios).

[0071] In one exemplary embodiment, such as Figure 5 As shown, the data quality management module performs quality verification on the scheduling data, which includes verifying the scheduling data according to a three-level verification dimension: field-level, record-level, and inter-table-level, based on the scheduling-specific data quality rule library. Specifically, this includes: field-level verification, which verifies the format, value range, length, and required fields of the fields; record-level verification, which verifies the total number of rows in a single table, data uniqueness, and logical relationships between fields; and inter-table-level verification, which verifies the primary and foreign key relationships, bidirectional data matching, and cross-table logical consistency between multiple data tables.

[0072] In some embodiments, the data quality management module is also used to generate a rectification work order and push it to the corresponding business responsible person when a verification anomaly occurs, receive the business responsible person's actions of issuing, processing, providing feedback, withdrawing or removing the rectification work order, and perform statistical analysis on the number and processing rate of the rectification work orders according to the business dimension and the time dimension.

[0073] To address the issues of fragmented existing data quality control systems that only support single-entry verification and lack closed-loop processes, this embodiment divides the verification into three layers and coordinates with a dedicated scheduling rule base (pre-setting six quality standards: standardization, completeness, consistency, accuracy, timeliness, and frequency, as well as three rule types: inter-table association rules, single-table record rules, and field rules). When verification anomalies occur, a rectification work order is automatically generated and pushed to the corresponding business responsible person. The responsible person then performs operations such as issuing, processing, providing feedback, withdrawing, or removing the rectification work order, realizing closed-loop management of data issues from discovery, dispatch, rectification to archiving, and performing statistical analysis on the processing quantity and processing rate according to business and time dimensions.

[0074] In one exemplary embodiment, the data resource management module is specifically used to store scheduling data into a five-level hierarchical storage architecture consisting of a raw library, a resource library, a theme library, a business library, and a model library.

[0075] The database consists of several modules: the raw database, which stores the incoming scheduling data as is and serves as the basis for data traceability; the resource database, which cleans, standardizes, and integrates the scheduling data in the raw database to form shared basic data across the entire domain, with storage modes including central storage and local storage to reduce cross-regional traffic; the theme database, which integrates data around core business themes such as load forecasting, power flow calculation, electricity spot market, and equipment monitoring; the business database, which stores dedicated data for specific scheduling business scenarios and works with the tag database for multi-dimensional tagging management; and the model database, which stores professional power scheduling data models according to a four-step modeling process of business modeling, conceptual modeling, logical modeling, and physical modeling.

[0076] To address the problems of chaotic and hierarchical data storage architecture in existing systems, which prevents traceability and reuse, this embodiment implements five-level hierarchical persistence based on processing depth and business attributes.

[0077] In one exemplary embodiment, the data asset management module further includes at least one of a data resource catalog subunit, a metadata management subunit, a data standard management subunit, and a data operation and maintenance subunit.

[0078] The data asset management module does not physically store the scheduling data. Instead, it uses the scheduling data stored hierarchically by the data resource management module as the object of control. It catalogs, reviews, publishes, and subscribes through the data resource catalog subunit. It collects and manages technical and business metadata (supporting version management, change notifications, difference comparisons, and historical retrospectives) through the metadata management subunit. It formulates unified format, naming, and definition standards and establishes a three-way mapping between standards, metadata, and quality rules. This transforms the scheduling data into manageable, shareable, and traceable data assets on the storage platform of the data resource management module. The data operation and maintenance subunit provides multi-dimensional monitoring and report output for resources, data, and services.

[0079] To address the issue that most power data systems only store data and lack the four major asset management capabilities, resulting in a lack of assetization, this embodiment transforms scheduling data into manageable, shareable, and traceable assets on top of the storage base.

[0080] It's important to clarify that the data resource management module and the data asset management module form a collaborative upper-lower layer relationship between storage foundation and asset governance: the data resource management module is responsible for scheduling the physical tiered storage of data, addressing the questions of where and how data is stored, thus providing the data resource foundation; the data asset management module itself does not physically store business data, but rather uses the tiered data stored by the data resource management module as the object of management. Through resource catalogs, metadata, data standards, and data operations and maintenance, it catalogs, describes, standardizes, and maintains resources, addressing the issues of how data is cataloged, understood, and governed, thereby transforming data residing at the resource layer into assets. The logical boundaries between the two are clear: the former focuses on storage, while the latter focuses on governance.

[0081] In an exemplary embodiment, the data value service module includes a directory service subunit, a call service subunit, and an intelligent retrieval service subunit. The directory service subunit catalogs, reviews, publishes, and subscribes to data resources and achieves secure sharing based on hierarchical permissions. The call service subunit builds a distributed data service bus based on the REST application programming interface and provides service routing, permission control, call rate limiting, and log auditing. The intelligent retrieval service subunit integrates natural language processing and knowledge graphs to perform unified semantic retrieval of structured and unstructured data.

[0082] To address the issues of existing platforms having limited data service capabilities and only providing basic queries, as well as the poor power safety adaptability and substandard cross-isolation transmission of general-purpose platforms, this embodiment constructs an integrated service system for directory subscription, interface calls, and intelligent semantic retrieval, and realizes one-way low-latency cross-regional service interaction based on physical isolation devices.

[0083] The aforementioned integrated platform / system for multi-source data fusion, quality control, and asset services in the power dispatch domain, through data exchange, quality governance, hierarchical storage, assetization, and data services in Security Zone 2 and Security Zone 3, connects multi-source data fusion, quality control, and asset services into an integrated data processing link. It can uniformly access multi-source heterogeneous dispatch data and achieve secure cross-zone transmission, thereby solving the problem that dispatch data is difficult to unify, effectively govern, and transform into externally serviceable assets, achieving the beneficial effect of improving the accuracy, consistency, and reusability of dispatch data.

[0084] In one exemplary embodiment, a data processing method for multi-source data fusion, quality control, and asset services in the power dispatch domain is provided, which is then applied to... Figure 2Taking the integrated system in the power dispatching domain as an example, the process includes the following steps: acquiring dispatching data from multiple heterogeneous data sources in the power dispatching domain; parsing and preprocessing the acquired dispatching data in the data exchange management module deployed in the second security zone; transmitting the processed dispatching data to the data quality management module, data resource management module, data asset management module, and data value service module deployed in the third security zone via a one-way file transfer method through the physical isolation device between the second and third security zones; performing quality verification on the dispatching data based on the data quality management module; storing the quality-verified dispatching data in layers according to processing depth and business attributes based on the data resource management module; constructing resource catalogs and metadata for the dispatching data stored in layers by the data resource management module based on the data asset management module, realizing the transformation of dispatching data from resources to assets; and providing standardized data services to upper-layer applications in the power dispatching domain based on the data value service module.

[0085] In one exemplary embodiment, such as Figure 3 As shown, the multi-source data processing method for the power dispatch domain in this application embodiment includes steps 302 to 314. Wherein:

[0086] Step 302, the data exchange step, that is, to obtain scheduling data from multi-source heterogeneous data sources in the power dispatch domain.

[0087] The multi-source heterogeneous data sources include at least one of the following: interface type, file type, database type, and message middleware type. Interface type data sources may include RESTful interfaces and WebService interfaces; file type data sources may include FTP, SFTP, XML, CSV, JSON, Excel, and PDF; database type data sources may include relational databases, non-relational databases, and columnar databases; message middleware type data sources may include Kafka and RocketMQ.

[0088] The system uses a data source management unit to uniformly configure various data sources, maintain basic information such as address, port, account, permissions and protocols, and support the addition, editing, deletion and status monitoring of data sources.

[0089] Optionally, the system adopts a plug-in architecture for its strategy configuration management unit, which adapts to structured, semi-structured, and unstructured data. It supports four collection modes: full, incremental, real-time, and batch. It also allows for visual configuration of data extraction, format conversion, field filtering, and data routing rules, and outputs data to local storage, various databases, message queues, or distributed file systems.

[0090] Step 304, the parsing and preprocessing step, involves parsing and preprocessing the acquired scheduling data in the data exchange management module deployed in the security zone 2.

[0091] Step 306, the cross-zone transfer step, involves transmitting the processed scheduling data to the data quality management module, data resource management module, data asset management module, and data value service module deployed in the security zone 3 via a one-way file transfer method through the physical isolation device between security zone 2 and security zone 3.

[0092] The data exchange management module is deployed in Security Zone 2, responsible for data collection, parsing, and initial transmission, and does not provide external business access. The data quality management module, data resource management module, data asset management module, and data value service module are deployed in Security Zone 3, providing services to business and operations personnel. Physical isolation devices are deployed between the two zones, allowing only unidirectional data transmission.

[0093] In one exemplary embodiment, such as Figure 4 As shown, the cross-regional shuttle step in step 306 includes steps 402 to 406. Wherein:

[0094] Step 402 involves employing a distributed architecture comprised of a scheduling center and an executor cluster to slice the scheduled data to be transmitted into tasks and execute the data transmission in a multi-threaded parallel manner. Optionally, the task scheduling management unit supports cluster fault tolerance and automatic failover, with scheduling methods including timed scheduling (CRON expression), event-triggered scheduling, serial / parallel / hybrid scheduling, Shell / SQL script scheduling, and cross-node proxy scheduling.

[0095] Step 404: During transmission, monitor server CPU, memory, JVM, node status, and data traffic. If the network or storage is interrupted, record the breakpoint location and resume transmission from the recorded breakpoint location after the network or storage is restored to avoid full retransmission. In some embodiments, the system can issue alerts via both SMS and email in case of anomalies, timeouts, or failures.

[0096] Step 406: Using a physical isolation device, the scheduling data is transferred from Security Zone 2 to Security Zone 3 via a one-way file transfer protocol. Security Zone 2 generates a standard data file, which is then transferred to Security Zone 3 via the SFTP protocol. Security Zone 3 automatically parses and stores the data.

[0097] Step 308, the quality verification step, involves performing quality verification on the scheduling data in the data quality management module, and generating a rectification work order to drive the closed-loop rectification of abnormal data when verification is abnormal.

[0098] In one exemplary embodiment, such as Figure 5 As shown, step 308, the three-layer verification and work order closure, includes steps 502 to 508. Wherein:

[0099] Step 502, field-level validation, validates the field's format, value range, length, and required fields;

[0100] Step 504, record-level validation, validates the total number of rows in a single table, data uniqueness, and logical relationships between fields;

[0101] Step 506, Inter-table level verification, verifies the primary and foreign key relationships, bidirectional data matching, and cross-table logical consistency between multiple data tables;

[0102] Step 508: When an anomaly is detected, a rectification work order is automatically generated and pushed to the corresponding business person in charge. The person in charge then performs operations such as issuing, processing, providing feedback, withdrawing, or removing the rectification work order, and the number and rate of processing are statistically analyzed according to business and time dimensions.

[0103] Based on the above embodiments, six quality standards and three rule types can be preset through a dedicated scheduling rule base, and rule version management can be supported; dataset management integrates data sources and data tables according to scheduling business, and establishes a data source-data table-business indicator mapping; quality management scheme orchestration binds verification rules and configures job scheduling strategies for different datasets.

[0104] Step 310, the tiered storage step, is to store the quality-verified scheduling data in the data resource management module in tiers according to processing depth and business attributes.

[0105] In an exemplary embodiment, the hierarchical storage architecture includes: a raw database that stores the accessed scheduling data as is and serves as the basis for data traceability, with retention periods (3 months / 1 year / long term) set according to business needs; a resource database that cleans, standardizes, and integrates the scheduling data in the raw database to form shared basic data across the entire domain, with storage modes including central storage and local storage; a thematic database that integrates data around the core business themes of power dispatching, supporting indicator calculation and multidimensional analysis; a business database that stores dedicated data for specific dispatching business scenarios, subdividing into business knowledge base, entity database, and thematic database, and using a tag database for multidimensional tagging management; and a model database that stores professional power dispatching data models according to a four-step modeling process of business modeling, conceptual modeling, logical modeling, and physical modeling.

[0106] Step 312, the assetization step, involves constructing a resource catalog and metadata for the scheduling data stored hierarchically in the data resource management module within the data asset management module, thereby transforming the scheduling data from resources into assets.

[0107] In an exemplary embodiment, the assetization steps may specifically include: data resource catalog construction, including unified cataloging, review, publishing, and subscription of resources such as databases, files, interfaces, and big data engines, and achieving orderly and secure sharing by combining hierarchical permissions; metadata management, including collecting technical metadata and business metadata, supporting addition, modification, version management, change notification, difference comparison, and historical retrospection, and automatically notifying subscribed users by email or SMS after metadata changes; data standard management, including formulating unified data format, naming, and definition standards to achieve a three-way mapping of standards, metadata, and quality rules; and data operation and maintenance, including providing resource monitoring, data monitoring, service monitoring, and report output.

[0108] Understandably, the data asset management module does not physically store the scheduling data. Instead, it uses the scheduling data stored in layers by the data resource management module as the object of control, and transforms the scheduling data into manageable, shareable, and traceable assets on its storage platform.

[0109] Step 314, the data service step, is to provide standardized data services to the upper-layer applications of the power dispatch domain in the data value service module.

[0110] In an exemplary embodiment, the data service steps specifically include: classifying data providers, data users, and service managers through a directory service, supporting data uploading, browsing, subscription, and permission configuration; building a distributed data service bus based on REST application programming interfaces through a call service, with synchronous deployment in main / branch centers, supporting multi-form data interaction such as messages, data streams, files, and audio / video, and possessing service routing, permission control, call rate limiting, and call log auditing capabilities; and integrating natural language processing and knowledge graph technologies through an intelligent retrieval service to support unified retrieval of structured and unstructured data and natural language semantic retrieval, reducing the threshold for scheduling data usage.

[0111] To better illustrate the integrated system and multi-source data processing method provided in the embodiments of this application, in one specific embodiment, the physical partition deployment of the integrated system includes:

[0112] Security Zone 2 is equipped with server clusters, data exchange modules, acquisition components, and SFTP transfer components; it only connects to the scheduling data network and the integrated data network, and does not allow external network access.

[0113] The three security zones deploy full-featured modules, database clusters, middleware, application services, and a front-end portal, providing access for internal network business personnel and operations and maintenance personnel.

[0114] Isolation equipment is deployed between Zone 2 and Zone 3, including forward and reverse physical isolation devices, strictly adhering to the requirements for horizontal electrical isolation, and prohibiting penetration access.

[0115] The data exchange management module (or data access layer) in Security Zone 2 serves as the system's unified data entry point. It is responsible for multi-source heterogeneous data access, policy configuration, distributed task scheduling, and data preprocessing. It can include four sub-units: data source management, policy configuration management, task scheduling management, and data acquisition. Each unit is described below:

[0116] (1) The data source management unit supports unified configuration of all types of scheduling data sources, specifically covering: interface types: Restful, WebService; file types: FTP, SFTP, XML, CSV, JSON, Excel, PDF; database types: relational databases, non-relational databases, columnar databases; message middleware: Kafka, RocketMQ, etc. The main functions of the data source management unit include: supporting the addition, editing, deletion, and status monitoring of data sources, and uniformly maintaining basic information such as address, port, account, permissions, and protocol.

[0117] (2) The strategy configuration management unit adopts a plug-in architecture, supports secondary development, and is compatible with structured, semi-structured, and unstructured data. Among them, the collection mode supports four collection modes: full, incremental, real-time, and batch, and is compatible with real-time scheduling data, static ledger data, and periodic report data; it supports visual configuration data extraction, format conversion, field filtering, and data routing rules; and it supports data output to local storage, various databases, message queues, and distributed file systems.

[0118] (3) The task scheduling management unit (distributed cluster scheduling) adopts a scheduling center + executor cluster architecture, which supports cluster fault tolerance and automatic fault switching. Among them, the scheduling methods can be timed scheduling, event-triggered scheduling, serial / parallel / hybrid scheduling, Shell / SQL script scheduling, and cross-node proxy scheduling; it supports automatic task slicing, multi-threaded parallel transmission, and breakpoint resume transmission; after network and storage interruption, it automatically records the breakpoint and resumes transmission without full retransmission; it can also monitor server CPU, memory, JVM, node status, and data traffic in real time, and send alarms through SMS and email channels in case of abnormality, timeout, or failure.

[0119] (4) Data acquisition unit, used to complete the entire process of data extraction, cleaning, reconciliation, and preprocessing. Specifically, it includes: Data parsing: supports database DML / DDL parsing, multi-format file parsing, and multi-protocol data reception; supports Push and Pull interaction modes; Data reconciliation: automatically verifies the number of data entries and field integrity after transmission, generates reconciliation logs, and archives abnormal data separately; Data cleaning: automatically filters missing, duplicate, and formatted data, and distinguishes between automatic and manual processing rules; Cross-regional transmission: transmission from Zone 2 to Zone 3 is carried out through physical isolation devices, using SFTP file transfer to meet the power cross-regional latency requirements.

[0120] The data quality management module (data governance layer) in the third security zone is used to build a dedicated data quality rule base for scheduling, a three-layer verification system, and a closed-loop governance system for work orders. It comprises seven sub-units: quality rules, datasets, quality solutions, multi-level verification, quality monitoring, issue tracking, and statistical analysis. Each unit is described below:

[0121] (1) Quality rule library, which is pre-set with power dispatch-specific rules, divided into six standards and three rule types, covering all dispatch business scenarios. Specifically, the six quality standards include standardization, completeness, consistency, accuracy, timeliness and frequency; the three rules include inter-table association rules, single-table record rules and field rules; and it also supports adding, enabling, disabling, searching and version management of rules, and can be expanded according to business iteration.

[0122] (2) Data set management: integrate data sources and data tables according to scheduling business (load forecasting, electricity spot market, equipment monitoring, etc.), establish the data source-data table-business indicator mapping relationship, and support the addition, editing and deletion of data sets.

[0123] (3) Quality management scheme arrangement: customize exclusive quality monitoring schemes for different datasets, bind verification rules, and configure job scheduling strategies (such as execution cycle, number of retries, routing strategy, timeout time, etc.).

[0124] (4) Three-level rule validation, including field-level validation: validating field format, value range, length, and required fields (such as mobile phone number, date, and code); record-level validation: validating the total number of rows in a single table, data uniqueness, and field logical relationships; and inter-table validation: validating primary and foreign key associations across multiple tables, bidirectional data matching, and cross-table logical consistency. In addition, the validation results record the erroneous fields, error reasons, and number of data entries in real time.

[0125] (5) Quality monitoring: Visually display the execution time, running status, and percentage of problem data for each monitoring scheme, and display the monitoring results by indicator dimension.

[0126] (6) Quality problem tracking (work order closed loop): After an anomaly is found during verification, a work order is automatically generated and pushed to the corresponding business person in charge; it supports the full process management of work order issuance, processing, feedback, withdrawal and removal, and realizes the closed loop of data problem discovery-dispatch-rectification-archiving.

[0127] (7) Monitoring problem statistical analysis can include two major statistical dimensions: business dimension according to business systems such as OMS, OCS, water regulation automation, etc., to count the total number of problems, the number of problems that have been processed, and the processing rate; time dimension to generate trend charts on a daily basis to intuitively show the problem change pattern and support governance optimization.

[0128] The data resource management module (data storage layer) in the third security zone adopts a five-level hierarchical storage architecture: raw database, resource database, theme database, business database, and model database. Data is persisted in layers according to the depth of data processing and business attributes, taking into account traceability, reuse, and business adaptation. The five-level hierarchical storage architecture is described below.

[0129] (1) The original database stores all the original data that is connected without any processing; the retention period (3 months / 1 year / long term) is set according to business needs and serves as the sole basis for data traceability; it supports data storage of all types.

[0130] (2) Resource library: cleans, standardizes and integrates the original data across sources to form basic data shared across the entire domain; there are two storage modes: the main center stores basic public data and manages it uniformly across the network; the local center stores massive incremental data and each branch center stores it locally to reduce cross-regional traffic.

[0131] (3) Thematic library, which integrates data around the core business themes of dispatching (load forecasting, power flow calculation, electricity spot market, equipment monitoring) to support indicator calculation and multi-dimensional analysis.

[0132] (4) Business database: Stores exclusive data for specific scheduling business scenarios, and subdivides it into business knowledge database, entity database and topic database; with supporting tag database, the data is managed in a multi-dimensional tagging manner to improve retrieval and reuse efficiency.

[0133] (5) Model library, which stores power dispatching professional data models and is designed according to the four-step modeling process: business modeling - conceptual modeling - logical modeling - physical modeling, to support dispatching professional algorithms and upper-level applications.

[0134] The data asset management module (data asset layer) in Security Zone 3 is used to transform scheduling data from resources to assets. It includes four sub-units: data resource catalog, metadata management, data standard management, and data operation and maintenance. These are described below.

[0135] (1) Data resource catalog: Unified cataloging, review, publication and subscription of resources such as databases, files, interfaces and big data engines; combined with hierarchical permissions, to achieve orderly and secure sharing of data resources.

[0136] (2) Metadata management, the scope of collection includes technical metadata and business metadata; core functions include: metadata addition, modification, version management, change notification, difference comparison, and historical retrospective; automatic email / SMS reminders to subscribed users after metadata changes.

[0137] (3) Data standard management: formulate unified data format, naming and caliber standards for power grid dispatch; realize the three-way mapping of standards, metadata and quality rules; support the addition, release, repeal, version traceability and document export of standards.

[0138] (4) Data operation and maintenance, realize full-dimensional visual operation and maintenance monitoring, including: resource monitoring such as CPU, memory, hard disk, cluster node load, etc.; data monitoring such as access traffic, inbound volume, backlog volume, data reconciliation rate, value rate, etc.; service monitoring such as interface call count, response time, number of exceptions, etc.; and automatic generation of daily / weekly / monthly operation and maintenance reports to support system optimization.

[0139] The data value service module (data service layer) in the third security zone is used to provide standardized and intelligent data services to the outside world, and to connect data with upper-layer scheduling applications. It includes three sub-units: directory service, call service, and intelligent retrieval. These are described below.

[0140] (1) Directory service, which can be divided into three types of roles to build a complete service system. The three types of roles include data providers, data users and service managers; the service system supports data uploading, browsing, subscription and permission configuration.

[0141] (2) Calling services can build a distributed data service bus based on REST API, with synchronous deployment of main / branch centers. It supports multi-form data interaction such as messages, data streams, files, and audio / video, and has the capabilities of service routing, permission control, call rate limiting, and call log auditing.

[0142] (3) Intelligent retrieval service can integrate natural language processing (NLP) and knowledge graph technology to break through the limitations of traditional keyword retrieval and support unified retrieval of structured and unstructured data; it supports natural language semantic retrieval, and business personnel can query data through daily descriptions; it can quickly eliminate data silos and improve data acquisition efficiency.

[0143] Based on the above partitioned deployment and the functional descriptions of each module, the step-by-step implementation process for each module includes:

[0144] Phase 1: Environment setup and basic integration.

[0145] Completed the deployment, network configuration, and physical isolation debugging of security zones 2 and 3; installed all middleware and development environment; and connected to the Idaas permission system, scheduling data network, and integrated data network to achieve account synchronization and data path connection.

[0146] Phase 2: Implementation of the data exchange management module.

[0147] Enter the data source for the entire network scheduling system and complete the configuration of protocols, accounts, and permissions; configure the collection strategy (real-time / incremental / batch) and conversion rules according to the business; deploy the distributed scheduling cluster and configure the scheduling cycle, alarm rules, and breakpoint resume parameters; conduct joint testing of data collection, cleaning, and reconciliation functions to verify the stability of cross-regional transmission.

[0148] Phase 3: Implementation of the data quality management module.

[0149] Import dedicated quality rules for scheduling and improve the rule base; divide datasets according to business functions and establish data source and data table mapping; configure quality monitoring schemes, three-layer verification rules, and job scheduling; debug work order issuance, processing, and statistical functions to verify the closed-loop process.

[0150] Phase 4: Implementation of the data resource management module.

[0151] Deploy a five-level database cluster, configure storage policies and data retention periods; debug the data flow chain in the following order: original database - resource database - theme database - business database - model database; configure tag database and multi-mode storage rules (centralized / local).

[0152] Phase 5: Implementation of the data asset management module.

[0153] Launch the data resource catalog, complete resource cataloging, publishing, and permission configuration; deploy metadata collection, version management, and change notification functions; formulate scheduling data standards and complete the mapping between standards and metadata and quality rules; launch a visual operation and maintenance monitoring panel.

[0154] Phase 6: Implementation of the data value service module.

[0155] Register the data service interface, configure routing, rate limiting, and permissions; launch the data directory portal and subscription function; deploy the NLP+knowledge graph retrieval engine and debug intelligent retrieval; connect to the scheduling upper-layer application and verify the effectiveness of service calls.

[0156] Phase 7: System-wide integration and testing.

[0157] Functional testing, performance testing, security testing, and stress testing were conducted, and after defects were fixed, the system entered trial operation.

[0158] The integrated system described in the above embodiments of this application can adopt a B / S architecture + microservices + distributed big data technology stack, and be developed based on a power grid-specific platform. The front end can use Vue, HTML5, and CSS3, while the back end can be implemented using open-source components such as distributed task scheduling, Canal data synchronization, RocketMQ message queues, Redis caching, and ElasticSearch retrieval. This integrated system can be divided into five layers from top to bottom: a data access layer (data exchange), a data governance layer (quality management), a data storage layer (resource management), a data asset layer (asset management), and a data service layer (value services), forming a complete closed loop of collection, fusion, governance, storage, assets, and services. The system features the following technological innovations: First, it employs a two-zone / three-zone collaborative and integrated architecture for power dispatching. Data exchange is deployed in the secure second zone, while governance, storage, assets, and services are deployed in the secure third zone, achieving unidirectional transmission via forward and reverse physical isolation devices, distributed cluster scheduling, and breakpoint resumption. Second, it establishes a three-layer data quality verification system (fields, records, and tables) and a work order closed-loop system. Third, it adopts a five-level hierarchical storage architecture comprising a raw database, resource database, theme database, business database, and model database. Finally, in the data service module, it enables the fusion of NLP / knowledge graph intelligent semantic retrieval. Through the integrated system described above, a full-link integrated data asset operation system encompassing data exchange, quality governance, hierarchical storage, asset management, and data services can be realized. This system can uniformly access multi-source heterogeneous data for dispatching and achieve secure cross-zone transmission, solving the problem of dispatching data being difficult to unify, effectively govern, and transform into externally serviceable assets.

[0159] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0160] The modules in the aforementioned integrated system for power dispatching can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can invoke and execute the corresponding operations of each module.

[0161] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores scheduling data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the multi-source data processing method for the power dispatch domain described in the preceding embodiment.

[0162] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the aforementioned method embodiment.

[0164] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned method embodiment.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the aforementioned method embodiments.

[0166] It should be noted that the collection, storage, transmission and processing of the data involved in this application (such as scheduling data, metadata, quality work orders, etc.) all comply with relevant regulations and security specifications. Data anonymization, code auditing and security crowdsourcing are performed throughout the development, testing and deployment process. All relevant sensitive data are encrypted or anonymized to meet the Level 3 protection requirements of the Information Security Protection System.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0169] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An integrated system for the power dispatch domain, characterized in that, The system includes a physical isolation device, a security zone 2 side module, and a security zone 3 side module, wherein: The physical isolation device is deployed between the power monitoring security zone 2 and security zone 3 and is used to realize unidirectional data transmission. The security zone 2 side module is deployed in the security zone 2 and the security zone 3 side module is deployed in the security zone 3. The security zone two side module includes: The data exchange management module is used to acquire scheduling data from multiple heterogeneous data sources in the power dispatch domain, parse and preprocess the scheduling data, and transfer the processed scheduling data unidirectionally to the three safety zones through the physical isolation device. The three-zone safety module includes: The data quality management module is used to perform quality verification on the scheduling data; The data resource management module is used to store quality-verified scheduling data in layers according to processing depth and business attributes. The data asset management module is used to build resource catalogs and metadata for the scheduling data of tiered storage; The data value service module is used to provide standardized data services to upper-layer applications in the power dispatch domain, based on the resource catalog and metadata constructed by the data asset management module.

2. The system according to claim 1, characterized in that, The data exchange management module includes at least one of the following: a data source management unit, a policy configuration management unit, a task scheduling management unit, and a data acquisition unit, wherein: The data source management unit is used to uniformly configure and monitor the status of multi-source heterogeneous data sources, including interface classes, file classes, database classes, and message middleware classes. The strategy configuration management unit is used to configure at least one of the following collection modes: full, incremental, real-time, and batch, as well as data extraction, format conversion, field filtering, and data routing rules. The data acquisition unit is used to parse, reconcile, and clean the scheduling data under the drive of the data source management unit and the policy configuration management unit. The task scheduling management unit is used to perform task slicing, multi-threaded parallel transmission, and breakpoint resumption scheduling control on the scheduled data to be transmitted, providing distributed scheduling capabilities for the one-way file transfer of the physical isolation device.

3. The system according to claim 1, characterized in that, The process by which the data quality management module verifies the quality of the scheduling data includes: Based on a dedicated data quality rule library for scheduling, the scheduling data is validated using a three-tiered verification system: field-level, record-level, and inter-table-level. Field-level validation verifies the format, value range, length, and required fields of a field. Record-level validation verifies the total number of rows in a single table, data uniqueness, and logical relationships between fields. Inter-table level validation verifies primary and foreign key relationships, bidirectional data matching, and cross-table logical consistency among multiple data tables; The data quality management module is also used to generate rectification work orders when verification anomalies occur and push them to the corresponding business responsible persons, receive the issuance, processing, feedback, withdrawal or removal operations performed by the business responsible persons on the rectification work orders, and perform statistical analysis on the number and processing rate of rectification work orders according to business and time dimensions.

4. The system according to claim 1, characterized in that, The data resource management module is specifically used for: The scheduling data is stored in a five-level hierarchical storage architecture consisting of a raw database, a resource database, a topic database, a business database, and a model database, wherein: The original library is used to store the accessed scheduling data as is and to serve as the basis for data traceability. The resource library is used to clean, standardize, and integrate the scheduling data in the original library to form a shared basic data across the entire domain. The subject library is used to integrate the dispatch data in a targeted manner around the core business themes of power dispatch; The business library is used to store exclusive data for specific scheduling business scenarios and to work with the tag library for multi-dimensional tagging management. The model library is used to store power dispatching professional data models according to the four-step modeling process of business modeling, conceptual modeling, logical modeling and physical modeling.

5. The system according to claim 1, characterized in that, The data asset management module further includes at least one of the following: a data resource catalog subunit, a metadata management subunit, and a data standard management subunit, wherein: The data asset management module does not physically store the scheduling data. Instead, it uses the scheduling data stored hierarchically by the data resource management module as the control object. The scheduling data is cataloged, reviewed, published, and subscribed to through the data resource catalog subunit. Technical metadata and business metadata are collected and managed through the metadata management subunit. A three-way mapping of standards, metadata, and quality rules is established through the data standard management subunit. In order to transform the scheduling data into controllable, shareable, and traceable data assets on the storage base of the data resource management module.

6. The system according to claim 1, characterized in that... The data value service module includes a directory service subunit, an invocation service subunit, and an intelligent retrieval service subunit, wherein: The directory service subunit is used to catalog, review, publish and subscribe to data resources, and to achieve secure data sharing based on hierarchical permissions; The call service subunit is built on a distributed data service bus based on the REST application programming interface, providing data interaction covering messages, data streams, files and audio and video, and has the capabilities of service routing, permission control, call rate limiting and call log auditing. The intelligent retrieval service subunit integrates natural language processing and knowledge graph technologies to perform unified semantic retrieval of structured and unstructured data.

7. A multi-source data processing method for the power dispatch domain, characterized in that, The method includes: Acquire scheduling data from multiple heterogeneous data sources in the power dispatch domain; The acquired scheduling data is parsed and preprocessed in the data exchange management module deployed in Security Zone 2; Through the physical isolation device between Security Zone 2 and Security Zone 3, the processed scheduling data is transmitted to the data quality management module, data resource management module, data asset management module and data value service module deployed in Security Zone 3 via a one-way file transfer method; The scheduling data is quality verified based on the data quality management module. Based on the data resource management module, the quality-verified scheduling data is stored in layers according to processing depth and business attributes; Based on the data asset management module, a resource catalog and metadata are constructed from the scheduling data stored hierarchically by the data resource management module, thereby realizing the transformation of scheduling data from resources to assets; Based on the data value service module, standardized data services are provided to upper-layer applications in the power dispatch domain.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method of claim 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method of claim 7.

Images