A system and method for generating an indicator data service and a tag data service

Abstract

The application discloses a kind of index data service and label data service generation system and method, including multi-source data acquisition module, for the preprocessing of offline side power grid business data and real-time side power grid business data obtains offline side detailed data and real-time side detailed data;Offline and real-time fusion calculation module, for the aggregation processing of offline side detailed data and real-time side detailed data obtains offline side aggregation data and real-time side aggregation data, and then time-space alignment processing obtains fusion data;Index and label management module, for the index calculation and double-mode label generation of fusion data obtains index result table and label portrait wide table;Data service encapsulation module, for the service interface encapsulation of index result table and label portrait wide table generates index data service and label data service.The application can realize the whole-link standardization management and control of power grid business data from acquisition, processing, storage to service, and can be widely applied in the field of data processing technology.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a system and method for generating indicator data services and tag data services. Background Technology

[0002] A data processing architecture for power grid business data can achieve unified aggregation and sharing of power grid business data, improving the efficiency of data application construction. However, existing data processing architectures built on components such as Kafka, MPP, Hive, and Flink suffer from problems such as poor consistency, low reusability, inconsistent indicator definitions, and weak tag timeliness in terms of data service standardization, indicator management, tag system construction, and data layered governance. Summary of the Invention

[0003] To address the aforementioned technical issues, the purpose of this application is to provide a system and method for generating indicator data services and tag data services, which can achieve standardized management and control of the entire chain of power grid business data from collection, processing, storage to service.

[0004] To achieve the above objectives, one aspect of this application proposes an indicator data service and tag data service generation system, the system comprising: The multi-source data acquisition module is used to acquire offline-side power grid business data and real-time-side power grid business data, and to preprocess the offline-side power grid business data and the real-time-side power grid business data to obtain offline-side detailed data and real-time-side detailed data. The offline and real-time fusion computing module is connected to the multi-source data acquisition module. It is used to aggregate the offline side detailed data and the real-time side detailed data to obtain offline side aggregated data and real-time side aggregated data. Then, it performs spatiotemporal alignment processing on the offline side aggregated data and the real-time side aggregated data to obtain fused data. The indicator and tag management module is connected to the offline and real-time fusion computing module. It is used to perform indicator calculation on the fusion data based on metadata configuration to obtain an indicator result table, and to generate dual-mode tags on the fusion data to obtain a tag profile wide table. The data service encapsulation module, connected to the indicator and tag management module, is used to encapsulate the service interfaces of the indicator result table and the tag profile wide table respectively, and generate indicator data services and tag data services.

[0005] In some embodiments, the system further includes: The hierarchical data storage module is connected to the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module. It is used to store the offline power grid business data, the real-time power grid business data, the offline detailed data, the real-time detailed data, the aggregated data, the indicator result table, the tag profile wide table, the indicator data service, and the tag data service.

[0006] In some embodiments, the system further includes: The intelligent task scheduling module is connected to the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module. It is used to receive task scheduling instructions triggered by users and control the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module according to the task scheduling instructions.

[0007] In some embodiments, the offline and real-time fused computing module includes: The basic computing layer is connected to the multi-source data acquisition module and is used to perform parallel aggregation processing on the offline side detailed data based on the MPP engine to obtain the offline side aggregated data, and to perform streaming aggregation processing on the real-time side detailed data based on Flink to obtain the real-time side aggregated data. The fusion scheduling layer, connected to the basic computing layer, is used to perform spatiotemporal alignment processing on the offline-side aggregated data and the real-time-side aggregated data by defining FlinkSQL functions to obtain the fused data. The real-time-side aggregated data is set to have a higher priority than the offline-side aggregated data. When the same aggregated data has both offline and real-time versions, the real-time version of the aggregated data is processed first.

[0008] In some embodiments, the metrics and tag management module includes: The indicator management submodule is connected to the offline and real-time fusion computing module. It is used to solidify the indicator caliber parameters into metadata, obtain the indicator metadata registry, and build an indicator task generation engine based on Flink and MPP. Then, based on the indicator task generation engine, it performs metadata configuration according to the indicator metadata registry and the aggregated data to obtain the indicator result table. The tag generation submodule, connected to the offline and real-time fusion computing module, is used to construct offline tags for the aggregated data based on Hive to obtain offline side tags, and to generate real-time tags for the aggregated data based on Flink to obtain real-time side tags. Then, the offline side tags and the real-time side tags are merged to obtain the tag profile wide table.

[0009] In some embodiments, the data service encapsulation module includes: The API service submodule is connected to the indicator and tag management module and is used to convert the indicator result table into an API interface to generate the indicator data service, and to convert the tag profile wide table into an API interface to generate the tag data service. The data push submodule, connected to the indicator and tag management module, is used to push the indicator data service and the tag data service to the target downstream application.

[0010] In some embodiments, the hierarchical data storage module includes: The detail layer, connected to the multi-source data acquisition module, is used to store the offline-side power grid business data, the real-time-side power grid business data, the offline-side detail data, and the real-time-side detail data; The aggregation layer, connected to the offline and real-time fusion computing module and the indicator and tag management module, is used to store the aggregated data, the indicator result table, and the tag profile wide table. The service layer, connected to the data service encapsulation module, is used to store the indicator data service and the tag data service.

[0011] To achieve the above objectives, another aspect of this application proposes a method for generating indicator data services and tag data services, the method comprising the following steps: The offline power grid business data and the real-time power grid business data are acquired through the multi-source data acquisition module. The offline power grid business data and the real-time power grid business data are preprocessed to obtain offline detailed data and real-time detailed data. The offline and real-time fusion computing module aggregates the offline and real-time detailed data to obtain offline aggregated data and real-time aggregated data. Then, it performs spatiotemporal alignment processing on the offline aggregated data and real-time aggregated data to obtain fused data. The indicator and tag management module calculates indicators on the fused data based on metadata configuration to obtain an indicator result table, and generates dual-mode tags on the fused data to obtain a wide tag profile table. The data service encapsulation module encapsulates the service interfaces of the indicator result table and the tag profile wide table to generate indicator data service and tag data service.

[0012] In some embodiments, the method further includes: The hierarchical data storage module stores the offline power grid business data, the real-time power grid business data, the offline detailed data, the real-time detailed data, the aggregated data, the indicator result table, the tag profile wide table, the indicator data service, and the tag data service.

[0013] In some embodiments, the method further includes: The intelligent task scheduling module receives task scheduling instructions triggered by the user and controls the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module according to the task scheduling instructions.

[0014] The beneficial effects of this application are as follows: The indicator data service and tag data service generation system and method of this application include a multi-source data acquisition module, an offline and real-time fusion calculation module, an indicator and tag management module, and a data service encapsulation module. The multi-source data acquisition module is used to acquire offline and real-time power grid business data, preprocess the offline and real-time power grid business data to obtain offline and real-time detailed data; the offline and real-time fusion calculation module is used to aggregate the offline and real-time detailed data to obtain offline and real-time aggregated data, and then perform spatiotemporal alignment processing on the offline and real-time aggregated data to obtain fused data; the indicator and tag management module is used to perform indicator calculation on the fused data based on metadata configuration to obtain an indicator result table, and generate dual-mode tags on the fused data to obtain a tag profile wide table; the data service encapsulation module is used to encapsulate the service interfaces of the indicator result table and the tag profile wide table respectively to generate indicator data services and tag data services. This application solves the problem of inconsistent time series of cross-source data by spatiotemporally aligning and fusing offline and real-time aggregated data. It also generates an indicator result table by calculating indicators based on metadata configuration and generates a wide label profile table based on a dual-mode label generation method of offline batch and real-time incremental updates. This can unify indicator caliber and improve label timeliness. Finally, the indicator result table and the wide label profile table are encapsulated into a unified shared service interface to realize the reuse of power grid business data. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments of this application are described below. It should be understood that the drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1This is a structural block diagram of an indicator data service and tag data service generation system provided in one embodiment of this application; Figure 2 A schematic diagram of the implementation environment of an indicator data service and tag data service generation system provided in one embodiment of this application; Figure 3 A structural block diagram of an indicator data service and tag data service generation system provided in another embodiment of this application; Figure 4 A schematic diagram illustrating the entire process from data collection to indicator services provided in one embodiment of this application; Figure 5 This application provides a full-link data flow diagram for an indicator data service and tag data service generation system according to one embodiment of the present application. Figure 6 This is a schematic diagram illustrating the steps of a method for generating indicator data service and tag data service according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0021] Kafka, also known as a distributed message queue or event streaming platform, is a high-throughput, low-latency distributed publish-subscribe messaging system. This component, based on partitioning and replication mechanisms, is used to achieve real-time, persistent data transfer between heterogeneous systems.

[0022] MPP: Massively Parallel Processing, also known as distributed parallel database architecture, is a database design concept that breaks down computational tasks into multiple independent nodes for parallel execution. Based on the divide-and-conquer principle, this architecture distributes data across different nodes, executes query segments simultaneously on each node, and then aggregates the results, achieving second-level or even millisecond-level query response times for massive datasets.

[0023] Hive, also known as a Hadoop data warehouse tool, is an offline data warehouse software based on the Hadoop ecosystem. This component provides a SQL-like query language (HiveQL), which translates user-submitted SQL statements into MapReduce, Tez, or Spark jobs at the underlying level, thereby enabling querying and analysis of massive amounts of structured data stored on HDFS (Hadoop Distributed File System).

[0024] Flink, also known as a distributed stream processing engine, is a true real-time stream computing framework. This component is based on an event-driven model, uses "unbounded streams" as its core data form, and supports real-time data processing with millisecond-level latency and high throughput.

[0025] A data processing architecture for power grid business data can achieve unified aggregation and sharing of power grid business data, improving the efficiency of data application construction. However, existing data processing architectures built on components such as Kafka, MPP, Hive, and Flink have the following technical shortcomings in terms of data service standardization, indicator management, tag system construction, and data layered governance, making it difficult to meet the core business requirements for data consistency, reusability, sharing, and reliability: I. Heterogeneous and fragmented batch and stream processing links pose technical bottlenecks to data consistency and reusability. In existing technologies, real-time streaming data carried by Kafka and offline batch data stored in Hive are processed in physically isolated, independent links. The lack of a unified data synchronization verification and version control mechanism easily leads to data timing misalignment and field semantic conflicts. Simultaneously, various business systems often adopt a siloed approach to developing dedicated data interfaces, resulting in heterogeneous interface protocols and data formats. This necessitates repeated extraction, transformation, and loading (ETL) operations on the same data; cross-system interface calls suffer from high redundancy and high coupling, leading to low data asset reuse rates.

[0026] Second, the lack of unified control over indicator calculation logic limits the efficiency and timeliness of data sharing. The existing indicator calculation module, built on Flink and MPP, lacks a standardized mechanism for managing indicator metadata and solidifying calculation logic. Different business lines define different statistical dimensions, time granularities, and filtering rules for the same indicator, leading to conflicting and incomparable calculation results. Furthermore, the indicator calculation results are not encapsulated into a unified shared service interface, requiring repeated development of calculation tasks for cross-departmental calls, resulting in wasted cluster computing resources. The coexistence of high latency in offline calculations and jitter in real-time calculations leads to insufficient overall data timeliness, affecting the accuracy of decision analysis.

[0027] Third, the tag computing and storage architecture is simplistic, resulting in poor performance in adapting to dynamic scenarios and handling high-concurrency queries. The existing tag system primarily relies on Hive for offline batch processing, leading to long tag update cycles, weak timeliness, and difficulty in integrating real-time behavioral data to support dynamic business scenarios. Furthermore, the underlying tag storage lacks hierarchical indexes and multi-dimensional aggregation design, resulting in significant performance bottlenecks when performing multi-dimensional tag combination queries. High interface response latency in high-concurrency scenarios fails to meet the real-time query requirements of business systems.

[0028] Fourth, the data tiered processing mechanism is imperfect, resulting in insufficient data integrity and end-to-end reliability. At the detail layer, existing Flink real-time cleaning tasks have limited ability to identify and automatically repair anomalous data (such as missing, duplicate, or formatted errors), making it difficult to guarantee the integrity of the original data. At the aggregation layer, distributed aggregation computation relying on the MPP engine often suffers from biased calculation results and high recalculation costs due to unreasonable data partitioning strategies or poor task resource scheduling.

[0029] In view of this, this application proposes an indicator data service and tag data service generation system, including a multi-source data acquisition module, an offline and real-time fusion calculation module, an indicator and tag management module, and a data service encapsulation module. The multi-source data acquisition module is used to acquire offline and real-time power grid business data, preprocess the offline and real-time power grid business data to obtain offline and real-time detailed data. The offline and real-time fusion calculation module is used to aggregate the offline and real-time detailed data to obtain offline and real-time aggregated data, and then perform spatiotemporal alignment processing on the offline and real-time aggregated data to obtain fused data. The indicator and tag management module is used to perform indicator calculation on the fused data based on metadata configuration to obtain an indicator result table, and generate dual-mode tags on the fused data to obtain a tag profile wide table. The data service encapsulation module is used to encapsulate the service interfaces of the indicator result table and the tag profile wide table respectively to generate indicator data services and tag data services. This application solves the problem of inconsistent time series of cross-source data by spatiotemporally aligning and fusing offline and real-time aggregated data. It also generates an indicator result table by calculating indicators based on metadata configuration and generates a wide label profile table based on a dual-mode label generation method of offline batch and real-time incremental updates. This can unify indicator caliber and improve label timeliness. Finally, the indicator result table and the wide label profile table are encapsulated into a unified shared service interface to realize the reuse of power grid business data.

[0030] Reference Figure 1 , Figure 1 This is a structural block diagram of an indicator data service and tag data service generation system provided in one embodiment of this application. This application proposes an indicator data service and tag data service generation system, which includes: The multi-source data acquisition module is used to acquire offline power grid business data and real-time power grid business data, and to preprocess the offline and real-time power grid business data to obtain offline detailed data and real-time detailed data. The offline and real-time fusion computing module is connected to the multi-source data acquisition module. It is used to aggregate the offline and real-time detailed data to obtain offline aggregated data and real-time aggregated data. Then, the offline aggregated data and real-time aggregated data are spatiotemporally aligned to obtain fused data. The indicator and tag management module is connected to the offline and real-time fusion computing module. It is used to perform indicator calculation on the fusion data based on metadata configuration, obtain the indicator result table, and generate dual-mode tags on the fusion data to obtain a wide tag profile table. The data service encapsulation module connects to the indicator and tag management module and is used to encapsulate the service interfaces of the indicator result table and the tag profile wide table respectively, generating indicator data services and tag data services.

[0031] It should be noted that, as Figure 2 The diagram shows the implementation environment of an indicator data service and tag data service generation system provided in one embodiment of this application. This embodiment relies on a data center to build a unified basic data foundation and adopts a layered architecture design, which includes a multi-source data acquisition module, an offline and real-time fusion computing module, an indicator and tag management module, and a data service encapsulation module. Each module is customized and collaborative based on Kafka, MPP, Hive, and Flink components to provide comprehensive data support for upper-layer business applications.

[0032] Specifically, the multi-source data acquisition module serves as the entry point for data into the data lake, integrating both offline batch ingestion and real-time streaming ingestion engines. It collects power grid business data from source-end business systems (including power grid management platforms, customer service platforms, operation scheduling platforms, and external data platforms) via Kafka message queues and persists this data to an MPP database or HDFS (distributed file system) for storage. On the offline side, a customized Hive full or incremental synchronization task based on Sqoop supports efficient ingestion of structured data into the lake according to a partitioning strategy, yielding detailed offline data. On the real-time side, a data acquisition channel is built based on KafkaConnect, connecting to real-time data such as business system logs and behavioral tracking points. It also includes a built-in data format validator to verify the field integrity and format validity of the ingested data, filtering out abnormal data and generating verification logs to obtain detailed real-time data.

[0033] In some optional embodiments, for structured data partitioning strategies, the update time of the business data table can be used as the partition key, employing the date partitioning format dt=yyyyMMdd, and generating an independent partition directory daily. For example, taking a MySQL customer transaction table as an example, it is synchronized to a Hive table in the data lake via Sqoop, with dt as the partitioning field. A full or incremental synchronization task is scheduled every morning, and the transaction data for that day is written to the independent partition corresponding to dt=20260523. During queries, data can be pruned by date partitioning to reduce the scan range and avoid massive full table scans, thereby achieving efficient ingestion of structured data into the data lake and partition-filtered queries.

[0034] Reference Figure 3 , Figure 3 The structural block diagram of the indicator data service and tag data service generation system provided in another embodiment of this application is shown. Further, as an optional implementation, the system includes: The hierarchical data storage module is connected to the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module. It is used to store offline power grid business data, real-time power grid business data, offline detailed data, real-time detailed data, aggregated data, indicator result table, tag profile wide table, indicator data service, and tag data service.

[0035] As a further optional implementation, the hierarchical data storage module includes: The detail layer, connected to the multi-source data acquisition module, is used to store offline power grid business data, real-time power grid business data, offline detail data, and real-time detail data. The aggregation layer connects to the offline and real-time fusion computing module and the indicator and tag management module, and is used to store aggregated data, indicator result tables, and wide tables of tag profiles. The service layer connects to the data service encapsulation module and is used to store indicator data services and tag data services.

[0036] Specifically, such as Figure 2 As shown, the hierarchical data storage module, based on network-level data center resources, constructs a three-tier data storage system to achieve hierarchical and tiered data management, including a detail layer, a summary layer, and a service layer. The detail layer uses Hive as the storage medium, partitioned by business domain, storing raw offline power grid business data, real-time power grid business data, and cleaned offline and real-time detail data, supporting full traceability. The summary layer is built on an MPP engine, storing aggregated data after fusion calculations, indicator result tables, and wide tables for tag profiles, optimizing distributed query performance and supporting second-level response times. The service layer is built on Redis to construct a tag and high-frequency indicator cache library, reducing the query pressure on the MPP engine and improving API service response speed.

[0037] As an optional implementation, the offline and real-time fused computing module includes: The basic computing layer connects to the multi-source data acquisition module and is used to perform parallel aggregation processing on offline detailed data based on the MPP engine to obtain offline aggregated data, and to perform streaming aggregation processing on real-time detailed data based on Flink to obtain real-time aggregated data. The fusion scheduling layer, connected to the basic computing layer, is used to perform spatiotemporal alignment processing on offline and real-time aggregated data using custom FlinkSQL functions to obtain fused data. The priority of real-time aggregated data is set to be higher than that of offline aggregated data. When there are offline and real-time versions of the same aggregated data, the real-time version of the aggregated data is processed first.

[0038] In some optional embodiments, to address the problem of disconnect between offline and real-time data in existing data processing architectures, this application proposes a two-layer computing engine architecture, including a basic computing layer and a fusion scheduling layer. The basic computing layer is used to perform offline computations, including partitioning, pruning, and pre-aggregation, on offline detailed data using the MPP engine to obtain offline aggregated data, thereby improving offline data computation efficiency. It also uses Flink to build streaming computing tasks to compute real-time detailed data, achieving second-level processing of real-time data such as user behavior. In the fusion scheduling layer, this application proposes a spatiotemporal alignment scheduling strategy. A custom FlinkSQL function is used to align the time dimension of offline Hive data with real-time Kafka data, resolving the issue of inconsistent time sequences across data sources. Simultaneously, a built-in data conflict resolution mechanism is implemented. When the same aggregated data has both offline and real-time versions, it is automatically merged according to the rule of real-time priority and offline as a fallback, and a version traceability log is recorded.

[0039] Specifically, for the partition pruning execution process, offline detailed data is stored in the data lake by constructing multi-level partitions according to date and business type. After receiving the summary calculation SQL task, the MPP engine first parses the filtering conditions and time range filtering conditions in the query statement: extracting the time filtering parameters carried by the query (such as statistical data on user electricity consumption from 20260501 to 20260523) and matching the detailed layer partition field dt; then matching the business filtering conditions (such as only statistical data on the power grid management platform) and matching the detailed layer biz_type partition field; then filtering based on the MPP engine metadata index, only loading the target partition directory that matches the filtering conditions, directly skipping all detailed partitions outside the interval, without scanning the full amount of detailed data; finally, only the pruned target partition detailed data is sent to each computing node of MPP to participate in subsequent aggregation, reducing disk I / O and memory loading data volume.

[0040] For the pre-aggregation execution process, after partitioning and pruning are completed and detailed data is loaded into the memory of the MPP compute nodes, distributed local pre-aggregation is performed, which consists of two steps: node-local pre-aggregation and global secondary aggregation. First, local pre-aggregation is performed: the MPP engine independently reads the pruned detailed data allocated to each compute node, and performs local grouping, summation, counting, and deduplication calculations according to preset aggregation dimensions (transformer area number, electricity customer ID, date, electricity category, voltage level, etc.), generating a node-local intermediate aggregation table. Massive detail rows are pre-compressed into a small number of intermediate aggregation results within the node, significantly reducing the amount of data transmitted over the network between nodes. Next, shuffle redistribution is performed: the local pre-aggregation results of each node are repartitioned according to the aggregation dimension hash, and local pre-aggregated data of the same dimension are aggregated to the same target compute node. Finally, global final aggregation is performed: after aggregation, the MPP engine overlays and summarizes multiple local pre-aggregated values ​​under the same dimension, outputting a standard summary layer-width table, which is directly stored in the data lake summary partition, without requiring full global aggregation of the original massive details.

[0041] For the spatiotemporal alignment scheduling strategy, for example, the business primary key is set to the order ID, and the alignment granularity is configured with a two-level time dimension of day plus hour. In the offline scheduling stage, Sqoop and Spark tasks are executed at 02:00 every day to generate a full-day Hive offline device operation baseline table with dt=20260522, which fully covers all device inventory data in operation, shutdown, and maintenance status for the day. Subsequently, the ready signal with dt=20260522 is sent to Flink. On the real-time link side, Flink continuously consumes the Kafka device telemetry data stream, with real-time messages carrying event_time=2026-05-23 10:35:22, and caches the voltage, current, power, and other measurement point data collected in real time on an hourly window basis. When alignment is triggered, Flink loads the custom time_dim_join() function, passing in the Hive offline device operation baseline table, the Kafka real-time order stream, and the alignment granularity parameters. The function first performs historical time period matching, loading all offline baselines for dt=20260522 as historical cumulative running data. Next, it performs real-time time period matching for the current day, filtering real-time telemetry data for each hour of event_time belonging to 20260523, and associating them with offline historical device records without corresponding real-time increments via device ID. When outputting the alignment result, under the same time and device primary key, the offline existing data carries the historical running baseline, while the real-time stream is overlaid with the newly added telemetry values ​​for the current day. After complete time-series alignment, the fused index is output. Correction alignment is performed the following day. After the offline baseline for dt=20260523 is generated in the early morning of May 24th, the scheduler automatically triggers realignment to correct any potential data loss or time-series errors in the real-time stream of May 23rd using complete offline partitioning.

[0042] As an optional implementation, the metrics and tag management module includes: The indicator management submodule is connected to the offline and real-time fusion computing module. It is used to solidify indicator caliber parameters into metadata, obtain the indicator metadata registry, and build an indicator task generation engine based on Flink and MPP. Then, based on the indicator task generation engine, the indicator metadata registry and aggregated data are used to configure the metadata and obtain the indicator result table. The tag generation submodule is connected to the offline and real-time fusion computing module. It is used to build offline tags for aggregated data based on Hive to obtain offline side tags, and to generate real-time tags for aggregated data based on Flink to obtain real-time side tags. Then, the offline side tags and real-time side tags are merged to obtain a wide tag profile table.

[0043] Specifically, addressing the issues of inconsistent indicator definitions and weak label timeliness, this application's embodiments construct a standardized indicator and label management module, including an indicator management submodule and a label production submodule, such as... Figure 4 The diagram illustrates the entire process from data collection to indicator services. The indicator management submodule designs an indicator metadata registry, solidifying parameters such as indicator calculation logic, statistical dimensions, time granularity, and filtering conditions into metadata to support cross-business line reuse. Simultaneously, it builds an indicator calculation task generation engine based on Flink and MPP, allowing users to automatically generate offline or real-time indicator calculation tasks by configuring metadata, avoiding redundant development. The tag production submodule employs a dual-mode tag production approach: offline batch generation and real-time incremental updates. The offline side uses Hive to build long-term stable offline tags, including basic customer attribute tags and electricity consumption characteristic tags, such as user category (residential / commercial / industrial), voltage level, contract capacity, and historical electricity consumption trends. The real-time side uses Flink stream computing to generate instantaneous dynamic real-time tags, such as transformer area load status tags and user real-time load tags. Finally, a tag fusion processor merges the dual-mode tags into a unified wide tag profile table, which is stored in the MPP engine to support high-concurrency queries.

[0044] As an optional implementation, the data service encapsulation module includes: The API service submodule connects to the indicator and tag management module and is used to convert indicator result tables into API interfaces to generate indicator data services, and to convert tag profile wide tables into API interfaces to generate tag data services. The data push submodule, connected to the indicator and tag management module, is used to push indicator data services and tag data services to target downstream applications.

[0045] Specifically, the data service encapsulation module serves as the unified outlet for data output, comprising an API service submodule and a data push submodule. The API service submodule encapsulates RESTful APIs based on Spring Cloud, transforming the metric result table and tag profile wide table into standardized API interfaces. It also designs interface caching strategies, setting cache expiration times for frequently queried interfaces to improve interface response performance. The data push submodule, based on DataX, customizes data integration tasks, pushing specified data tables to downstream application databases according to business needs. It also incorporates a built-in data synchronization verification mechanism to ensure data consistency before and after the push.

[0046] For example, taking the average power outage time metric as an example, business users can directly generate metric analysis results by dragging and dropping on the page, and generate analysis reports as needed. They can also apply to integrate the corresponding analysis results into the application as needed, forming an API service for the metric.

[0047] Reference Figure 3 As an optional implementation, the system further includes: The intelligent task scheduling module is connected to the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module. It is used to receive task scheduling instructions triggered by users and control the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module according to the task scheduling instructions.

[0048] Specifically, addressing the issues of disordered task scheduling and chaotic dependencies in existing systems, this application's embodiments design a visualized intelligent task scheduling module. This module, acting as the core control hub, receives user-configured task dependencies and scheduling instructions: it issues synchronization task instructions to the multi-source data acquisition module, computation task execution instructions to the offline and real-time fusion computing module, and metadata configuration instructions to the indicator and tag management module. It also collects task execution status and data quality verification logs from each module, generates visualized monitoring reports, and supports automatic alarms for abnormal situations. Furthermore, this module introduces a dynamic task priority adjustment strategy. When cluster resources are strained, it automatically prioritizes real-time indicator computation tasks to ensure the timeliness of core business data. Simultaneously, it incorporates a built-in task failure retry mechanism, automatically triggering retries for failed computation tasks and generating alarm logs.

[0049] The structure and function of the indicator data service and tag data service generation system of this application embodiment have been described in summary, such as... Figure 5 The diagram shows the complete data flow of the indicator data service and tag data service generation system. The following section explains the data flow and processing mechanism of the system and the interaction between system components.

[0050] Regarding data flow and processing mechanisms: First, the system collects structured and unstructured data from the source business system via a Kafka message queue and persists it to an MPP (Massively Parallel Processing) database or HDFS (Distributed File System) for storage. Second, for offline batch data, the system uses MPPSQL and HiveSQL computing engines for deep processing and aggregation calculations. For real-time data with high timeliness requirements, the system introduces the Flink real-time computing engine combined with Kafka message middleware for spatiotemporal alignment scheduling, thereby generating indicator result tables and tag profile wide tables. Subsequently, the system encapsulates the above data tables into a standardized data service, effectively shielding the differences in underlying heterogeneous storage and data structures, and provides a unified data supply capability to the outside world through a RESTful API interface, supporting efficient calls from upper-layer applications.

[0051] Regarding the interaction between system components: Data originates from various business systems, and after the multi-source data acquisition module obtains structured and unstructured data from each business system, it is entered into the lake after offline batch verification and real-time stream verification, and stored in Hive at the detail layer; then it enters the offline and real-time fusion computing module, which aggregates offline MPP and performs real-time Flink computing, and after spatiotemporal alignment fusion, the fusion result falls into the summary layer MPP storage; on this basis, the indicator and tag management module completes metadata registration, dual-mode tag generation, and profile wide table generation in sequence, and loads the generated indicator result table and tag profile wide table into the service layer Redis cache; finally, the data service encapsulation module delivers the data to the downstream business systems through API publishing or data push.

[0052] The structure, function, and working principle of the indicator data service and tag data service generation system of this application embodiment have been described above. It can be recognized that, compared with existing data processing architectures, the embodiments of this application have the following advantages: First, by using a custom Flink SQL function to perform spatiotemporal alignment scheduling of offline and real-time aggregated data, and setting priorities for data conflicts, it is possible to achieve seamless integration of Hive offline data and Kafka real-time data, solving the problems of inconsistent time sequences and version conflicts between cross-source data.

[0053] Second, by using the indicator metadata registry and the computing task generation engine that automatically generates offline or real-time computing tasks based on metadata, an indicator result table is generated, which can effectively unify the indicator criteria and realize the solidification and reuse of indicator logic.

[0054] Third, a wide table of tag profiles is generated based on a dual-mode tag generation method of offline batch and real-time incremental updates. The offline and real-time tags are merged through a tag merging device, which can improve the timeliness of tags, increase tag dimensions, and achieve unified management of dual-version tags.

[0055] Fourth, the introduction of a visual task dependency configuration parsing method, and dynamic scheduling based on business priority and resource status, can improve cluster resource utilization and solve the problems of disordered task scheduling and high latency of core business data.

[0056] Reference Figure 6 , Figure 6 This is a schematic diagram illustrating the steps of a method for generating indicator data services and tag data services according to an embodiment of this application. This embodiment provides a method for generating indicator data services and tag data services, executed by the aforementioned indicator data service and tag data service generation system, including the following steps S101 to S104: Step S101: Obtain offline power grid business data and real-time power grid business data through the multi-source data acquisition module, preprocess the offline power grid business data and real-time power grid business data to obtain offline detailed data and real-time detailed data. Step S102: Through the offline and real-time fusion computing module, the offline side detailed data and the real-time side detailed data are aggregated to obtain offline side aggregated data and real-time side aggregated data. Then, the offline side aggregated data and the real-time side aggregated data are spatiotemporally aligned to obtain fused data. Step S103: Through the indicator and tag management module, the indicator is calculated on the fused data based on the metadata configuration to obtain the indicator result table, and dual-mode tags are generated on the fused data to obtain the tag profile wide table. Step S104: Through the data service encapsulation module, service interfaces are encapsulated for the indicator result table and the tag profile wide table to generate indicator data service and tag data service.

[0057] As an optional further implementation, the method may also include the following step S105: Step S105: Store offline power grid business data, real-time power grid business data, offline detailed data, real-time detailed data, aggregated data, indicator result table, tag profile wide table, indicator data service, and tag data service through the hierarchical data storage module.

[0058] As an optional further implementation, the method may also include the following step S106: Step S106: Receive task scheduling instructions triggered by the user through the intelligent task scheduling module, and control the multi-source data acquisition module, offline and real-time fusion computing module, indicator and tag management module, and data service encapsulation module according to the task scheduling instructions.

[0059] The content of the above system embodiments is applicable to this method embodiment. The specific functions implemented in this method embodiment are the same as those in the above system embodiments, and the beneficial effects achieved are also the same as those achieved in the above system embodiments.

[0060] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer-readable storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0061] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0062] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0063] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A system for generating indicator data services and tag data services, characterized in that, The system includes: The multi-source data acquisition module is used to acquire offline-side power grid business data and real-time-side power grid business data, and to preprocess the offline-side power grid business data and the real-time-side power grid business data to obtain offline-side detailed data and real-time-side detailed data. The offline and real-time fusion computing module is connected to the multi-source data acquisition module. It is used to aggregate the offline side detailed data and the real-time side detailed data to obtain offline side aggregated data and real-time side aggregated data. Then, it performs spatiotemporal alignment processing on the offline side aggregated data and the real-time side aggregated data to obtain fused data. The indicator and tag management module is connected to the offline and real-time fusion computing module. It is used to perform indicator calculation on the fusion data based on metadata configuration to obtain an indicator result table, and to generate dual-mode tags on the fusion data to obtain a tag profile wide table. The data service encapsulation module, connected to the indicator and tag management module, is used to encapsulate the service interfaces of the indicator result table and the tag profile wide table respectively, and generate indicator data services and tag data services.

2. The system according to claim 1, characterized in that, The system also includes: The hierarchical data storage module is connected to the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module. It is used to store the offline power grid business data, the real-time power grid business data, the offline detailed data, the real-time detailed data, the aggregated data, the indicator result table, the tag profile wide table, the indicator data service, and the tag data service.

3. The system according to claim 1, characterized in that, The system also includes: The intelligent task scheduling module is connected to the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module. It is used to receive task scheduling instructions triggered by users and control the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module according to the task scheduling instructions.

4. The system according to claim 1, characterized in that, The offline and real-time fusion computing module includes: The basic computing layer is connected to the multi-source data acquisition module and is used to perform parallel aggregation processing on the offline side detailed data based on the MPP engine to obtain the offline side aggregated data, and to perform streaming aggregation processing on the real-time side detailed data based on Flink to obtain the real-time side aggregated data. The fusion scheduling layer, connected to the basic computing layer, is used to perform spatiotemporal alignment processing on the offline-side aggregated data and the real-time-side aggregated data by defining FlinkSQL functions to obtain the fused data. The real-time-side aggregated data is set to have a higher priority than the offline-side aggregated data. When the same aggregated data has both offline and real-time versions, the real-time version of the aggregated data is processed first.

5. The system according to claim 1, characterized in that, The indicator and label management module includes: The indicator management submodule is connected to the offline and real-time fusion computing module. It is used to solidify the indicator caliber parameters into metadata, obtain the indicator metadata registry, and build an indicator task generation engine based on Flink and MPP. Then, based on the indicator task generation engine, it performs metadata configuration according to the indicator metadata registry and the aggregated data to obtain the indicator result table. The tag generation submodule, connected to the offline and real-time fusion computing module, is used to construct offline tags for the aggregated data based on Hive to obtain offline side tags, and to generate real-time tags for the aggregated data based on Flink to obtain real-time side tags. Then, the offline side tags and the real-time side tags are merged to obtain the tag profile wide table.

6. The system according to claim 1, characterized in that, The data service encapsulation module includes: The API service submodule is connected to the indicator and tag management module and is used to convert the indicator result table into an API interface to generate the indicator data service, and to convert the tag profile wide table into an API interface to generate the tag data service. The data push submodule, connected to the indicator and tag management module, is used to push the indicator data service and the tag data service to the target downstream application.

7. The system according to claim 2, characterized in that, The hierarchical data storage module includes: The detail layer, connected to the multi-source data acquisition module, is used to store the offline-side power grid business data, the real-time-side power grid business data, the offline-side detail data, and the real-time-side detail data; The aggregation layer, connected to the offline and real-time fusion computing module and the indicator and tag management module, is used to store the aggregated data, the indicator result table, and the tag profile wide table. The service layer, connected to the data service encapsulation module, is used to store the indicator data service and the tag data service.

8. A method for generating indicator data services and tag data services, executed by the indicator data service and tag data service generation system as described in any one of claims 1 to 7, characterized in that, The method includes the following steps: The offline power grid business data and the real-time power grid business data are acquired through the multi-source data acquisition module. The offline power grid business data and the real-time power grid business data are preprocessed to obtain offline detailed data and real-time detailed data. The offline and real-time fusion computing module aggregates the offline and real-time detailed data to obtain offline aggregated data and real-time aggregated data. Then, it performs spatiotemporal alignment processing on the offline aggregated data and real-time aggregated data to obtain fused data. The indicator and tag management module calculates indicators on the fused data based on metadata configuration to obtain an indicator result table, and generates dual-mode tags on the fused data to obtain a wide tag profile table. The data service encapsulation module encapsulates the service interfaces of the indicator result table and the tag profile wide table to generate indicator data service and tag data service.

9. The method according to claim 8, characterized in that, The method further includes: The hierarchical data storage module stores the offline power grid business data, the real-time power grid business data, the offline detailed data, the real-time detailed data, the aggregated data, the indicator result table, the tag profile wide table, the indicator data service, and the tag data service.

10. The method according to claim 8, characterized in that, The method further includes: The intelligent task scheduling module receives task scheduling instructions triggered by the user and controls the multi-source data acquisition module, the offline and real-time fusion computing module, the indicator and tag management module, and the data service encapsulation module according to the task scheduling instructions.

Images