A curve calculation method and system based on data flow orchestration

Abstract

The application discloses a kind of based on data stream orchestration's curve calculation method and system, it is related to data processing technical field, the specific steps of this method are as follows: receiving time series curve data by distributed message bus, real-time monitoring, aggregation calculation and process scheduling are carried out using data stream automation tool, then raw data and calculation result are stored to columnar database synchronously, finally, through standard protocol, service access is provided, to support business decision, the application cooperates with distributed message bus, columnar storage database by data stream automation tool, constructs whole-process automation system, solves traditional data processing fragmentation problem, realizes data same warehouse storage and efficient traceability;At the same time, combined with real-time and batch calculation complementary design and standardized application interaction, meet business immediacy and depth analysis demand, simplify system deployment, provide flexible and convenient technical scheme for curve data processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a curve calculation method and system based on data stream orchestration. Background Technology

[0002] In fields such as power system operation monitoring and industrial equipment condition management, curve data, as time-series data with time as the horizontal axis and physical quantities or indicators as the vertical axis, is the core basis for business decision-making and condition assessment. With the development of IoT technology, the scale of curve data generated by external acquisition devices continues to expand, placing higher demands on the real-time performance, completeness, and collaboration of data processing. Currently, the industry generally adopts distributed message buses for data transmission, combined with data computing tools and storage databases to build a processing system. This aims to support key business operations such as power distribution area line loss statistics and equipment operation parameter early warning through data flow and analysis, meeting the practical needs of various fields for efficient utilization of time-series data.

[0003] In traditional curve data processing technologies, data access, transmission, computation, storage, and application interaction rely heavily on independently deployed devices or systems. The lack of a unified, automated collaborative mechanism between these stages necessitates manual intervention to configure data links and transformation rules, resulting in cumbersome and error-prone processes. Furthermore, raw data and computation results are typically stored in different databases, lacking unified time stamps and correlation identifiers. Data tracing and correlation queries require cross-system integration, leading to low efficiency. In addition, traditional computation methods often focus on single real-time or batch processing, making it difficult to simultaneously address the needs of immediate business response and in-depth historical data analysis. Moreover, application services require the development of customized interfaces to connect with storage systems, increasing system deployment and maintenance costs and failing to meet the actual needs of efficient curve data processing across various scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a curve calculation method and system based on data flow orchestration. Through deep integration of data flow automation tools with a distributed message bus and columnar storage database, a fully automated processing system of "data access - orchestration calculation - dual data storage" is constructed. This system solves the problems of fragmented processes and low efficiency in traditional curve data processing, achieving simultaneous storage of raw data and calculation results in the same database and efficient traceability. Simultaneously, by combining real-time and batch calculations, it meets the needs of business immediacy and in-depth analysis, simplifies system deployment, and provides flexible and convenient technical support for the efficient processing of curve data in multiple fields such as power and industry.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, a curve calculation method based on data stream orchestration, the specific steps of which are as follows:

[0006] Step 1, Data Access: Receive curve data reported in real time from external sources through a distributed message bus. The curve data is time-series data with time as the horizontal axis and a specific physical quantity or index as the vertical axis.

[0007] Step 2, Data Flow Orchestration and Calculation: By using a data flow automation tool to monitor the target data topic of the distributed message bus, the curve data is acquired in real time. The built-in aggregation calculation component of the data flow automation tool is used to aggregate and calculate the curve data according to preset business rules to obtain the calculation results.

[0008] Step 3, Dual Data Synchronization Storage: Through the dual data synchronization mechanism of the data flow automation tool, the original curve data and calculation results are synchronously written into the columnar storage database for storage in the same database, carrying a unified time stamp and association identifier.

[0009] Step 4, Data Application Interaction: The application service accesses the columnar storage database through standard network protocols to obtain raw curve data or calculation results to support business decisions and applications.

[0010] Furthermore, the distributed message bus in step 1 is Kafka, and the curve data includes voltage, current, and power time-series data in the power system or operating parameter time-series data of industrial equipment. Kafka is deployed in a cluster to transmit curve data.

[0011] Furthermore, the data flow automation tool in step 2 is Apache NiFi. The aggregation calculation includes numerical statistical calculation and time series feature extraction. The numerical statistical calculation includes average value, peak value, valley value, and summation. The time series feature extraction includes trend change rate and periodicity analysis. When this method is applied to the scenario of power distribution area line loss calculation, the aggregation calculation also includes line loss rate calculation. The line loss rate is calculated as the difference between the power supply and the sales volume, divided by the power supply and then multiplied by 100%. The power supply refers to the total electrical energy transmitted by the power system to the target distribution area, and the sales volume refers to the total electrical energy actually consumed by all users in the target distribution area.

[0012] Furthermore, in step 2, the data flow automation tool achieves real-time scheduling of the entire process through a built-in process scheduling component. The process scheduling component uses a priority queue to allocate data processing priorities and allocates computing resources through a dynamic load balancing mechanism to process curve data.

[0013] Furthermore, the unified time stamp in step 3 is the collection timestamp of the curve data, and the association identifier includes the data source identifier; the columnar storage database is ClickHouse, which is deployed through a sharded cluster and indexed by the time stamp and association identifier to store the original curve data and calculation results.

[0014] Furthermore, the standard network protocol in step 4 is the HTTP protocol, and the application service directly initiates query requests to the columnar storage database through the HTTP protocol; the data application interaction also includes a microservice collaboration mechanism, that is, request routing, user authentication and authorization and network attack protection are realized through the microservice gateway, and automatic registration and node discovery of application services are realized through the microservice registry center.

[0015] Furthermore, the method also includes a batch computing step: reading historical curve data or periodic statistical data through a distributed computing framework, performing deep mining and batch analysis to obtain batch analysis results, and synchronizing the batch analysis results to a columnar storage database or a distributed data warehouse; the distributed computing framework is Spark, and the distributed data warehouse is Hive.

[0016] On the other hand, a curve calculation system based on data stream orchestration includes:

[0017] Data access module: Employs a distributed message bus to receive curve data reported in real time by external acquisition devices and transmit the curve data to the data stream orchestration and calculation module; the curve data is time-series data with time as the horizontal axis and a specific physical quantity or index as the vertical axis;

[0018] Data Flow Orchestration and Computation Module: Employs automated data flow tools, including a data monitoring unit, an aggregation and computation unit, and a process scheduling unit. The data monitoring unit monitors the target data topics of the distributed message bus and acquires curve data in real time. The aggregation and computation unit uses the built-in aggregation and computation component to perform aggregation and computation on the curve data according to preset business rules to obtain the calculation results. The process scheduling unit schedules the data processing flow to achieve automated data flow from acquisition to computation.

[0019] Dual data synchronization storage module: includes a columnar storage database for storing raw curve data and calculation results, and the raw curve data and calculation results carry a unified time stamp and association identifier;

[0020] Data application interaction module: includes application service unit, which is used to access columnar storage database through standard network protocol to obtain raw curve data or calculation results to support business decision-making and application display.

[0021] Compared with existing technologies, this curve calculation method and system based on data stream orchestration has the following advantages:

[0022] I. This invention, through deep collaboration between data flow automation tools, distributed message buses, and columnar storage databases, constructs a fully automated system encompassing "data access - orchestration and computation - dual data storage," solving the problems of fragmented processes and excessive manual intervention in traditional curve data processing. Relying on the built-in aggregation computation component and dual-path data synchronization mechanism of the data flow automation tool, raw curve data and computation results can be stored in the same database with unified time stamps and association identifiers. Data traceability and correlation queries can be completed without cross-system integration, significantly simplifying the data processing chain. Simultaneously, through the priority queue and dynamic load balancing mechanism of the process scheduling component, efficient flow and computation of massive real-time curve data are ensured, avoiding data backlog. This adapts to the real-time data processing needs of multiple fields such as power and industry, improving the versatility and applicability of the technical solution.

[0023] II. This invention overcomes the limitations of traditional technologies in scenario adaptation and application integration through a complementary design of real-time and batch computing and a standardized application interaction scheme. The real-time computing stage relies on automated data flow tools to achieve instant data processing, meeting the immediacy requirements of business decisions. The batch computing step expands the processing capabilities of historical and periodic data through a distributed computing framework, achieving full-scenario data coverage. Application services directly access the columnar storage database through standard network protocols, combined with a microservice collaboration mechanism, enabling data interaction without the need to develop customized interfaces, thus reducing application integration costs. This design ensures both the response efficiency of real-time business and the in-depth needs of offline analysis, while simplifying system deployment and maintenance processes, providing more flexible and convenient technical support for the efficient utilization of curve data.

[0024] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0026] Figure 1 Here is a flowchart of the overall curve calculation method;

[0027] Figure 2 A detailed flowchart of data flow orchestration and computation;

[0028] Figure 3This is a system module architecture diagram. Detailed Implementation

[0029] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0030] Example 1:

[0031] This embodiment takes the scenario of real-time monitoring of line loss in power distribution areas and analysis of historical data in a power system as an example to explain in detail the specific implementation of the present invention. The hardware equipment, software tools and parameter configurations used are all practical and implementable solutions to ensure the repeatability and practicality of the technical solution.

[0032] like Figure 3 As shown, this system constructs a complete automated processing architecture through the collaborative efforts of four major modules: data access, orchestration and computation, storage, and application interaction. The system deployment and configuration are as follows:

[0033] Data Access Module: A distributed message bus cluster is built using Kafka 3.5.1, deployed with three nodes. The server configuration includes a 16-core CPU, 32GB of memory, and a 1TB SSD. A data topic named `power_line_loss_data` is created, configured with two partitions and two replicas to ensure data transmission reliability. External data acquisition devices are smart meters and transformers in the distribution area. Time-series data such as voltage, current, power, and cumulative energy are pushed to the Kafka cluster via the MQTT 3.1.1 protocol. The data format is JSON, and each data entry includes fields such as the acquisition timestamp, distribution area number, meter number, three-phase voltage value, three-phase current value, total active power, and cumulative energy value. Simultaneously, the system automatically generates corresponding indicator curves for the electrical parameters based on this 1-minute granularity time-series data: with the acquisition timestamp as the horizontal axis, the curves for the three-phase voltage value, three-phase current value, total active power, and cumulative energy value are plotted separately. The cumulative energy value is used as the vertical axis to form continuous indicator curves of "time-voltage", "time-current", "time-total active power", and "time-cumulative energy" at the single meter level. At the same time, the total active power data and cumulative energy data of all meters under the same distribution area are aggregated to generate indicator curves of "time-total active power" and "time-cumulative total energy" at the distribution area level. The subsequent line loss rate calculation relies on the interval integration result of the indicator curve or the time period difference of the "time-cumulative energy" indicator curve to obtain the power supply and sales volume of the corresponding time period, and then completes the calculation through the line loss rate formula.

[0034] Data stream orchestration and computation module: An Apache NiFi 1.23.0 cluster with one master and two slave nodes is constructed. Both master and slave nodes are configured with 24 CPU cores, 64GB of memory, and 2TB SSDs. A data stream processing flow is created within NiFi. The data listening unit uses the KafkaConsumer processor, configuring the Kafka cluster address and topic name. Based on the NiFi thread configuration, the batch data retrieval volume is set to 50,000 records per retrieval. The aggregation and computation unit uses the CalculateRecord processor, with two preset types of calculation rules: first, anomaly rule judgment for electrical parameters, based on the indicated curves, matching rules to the real-time curve characteristics of voltage, current, power, and energy to trigger anomaly flags; second, line loss rate calculation, for power distribution area scenarios, where power supply and sales can be obtained through three methods—

[0035] Method 1 involves performing time integration over a 1-minute interval on the "time-total active power" indication curve at the transformer substation level and the "time-total active power" indication curves of all user meters within the transformer substation.

[0036] Method 2 involves extracting the difference between the beginning and end of the "time-cumulative total energy" indicator curve at the transformer substation level and the "time-cumulative energy" indicator curve at the user side during the corresponding time period.

[0037] Method three involves directly reading the real-time recorded value of the "cumulative energy increment" field in the cumulative energy indicator curve. All three methods can obtain the power supply and sales volume of the transformer area during that period. Finally, the real-time line loss rate is calculated using the formula: line loss rate = (power supply - sales volume) / power supply × 100%. The process scheduling unit uses the NiFi built-in LoadBalancer controller and configures a priority queue. The line loss rate calculation task based on the energy indicator curve has a higher priority than ordinary statistical tasks. The calculation task is distributed to two slave nodes through a dynamic load balancing mechanism to achieve task load sharing.

[0038] Storage Module: A columnar storage database cluster is built using ClickHouse version 23.10, deployed with two shards, each containing one replica. The server configuration includes a 24-core CPU, 64GB of memory, and a 4TB SSD. Two data tables are created: `raw_power_data` stores raw curve data and basic data for the power consumption curve. Fields include collection timestamp, transformer area number, meter number, voltage value, current value, power value, cumulative power value, single meter cumulative power consumption point value, and time interval identifier for the power consumption curve. The collection timestamp is of type DateTime64, the transformer area number and meter number are of type String, and the voltage, current, and power values ​​are of type Float64. `calculated_line_loss_data` stores calculation results and derived data from the power consumption curve. Fields include collection timestamp, transformer area number, average voltage, peak current, total power, line loss rate, and aggregated value of cumulative total power consumption for the transformer area. The data types of each field correspond to those in the raw data table. Both tables are indexed using a composite index consisting of "collection timestamp + transformer area number + indicator curve time interval identifier" to ensure efficient querying of indicator curve data. Hive version 3.1.3 is deployed as a distributed data warehouse to store historical curve data. The data table structure is updated to include fields for single-meter cumulative energy indicator point values ​​and transformer area cumulative total energy indicator aggregate values, with data stored in partitions by month and transformer area number for easy batch retrieval of historical data.

[0039] Application Interaction Module: This module utilizes a Spring Cloud microservice architecture to build application services, including a line loss monitoring dashboard, data query interface, and an anomaly warning module. The microservice gateway is Spring Cloud Gateway, configured with routing rules mapping to each application service. The microservice registry uses Nacos version 2.3.2 to achieve automatic registration and discovery of application services. Application services directly initiate SQL query requests to the ClickHouse cluster via the HTTP 1.1 protocol, supporting filtering of raw data or calculated results by transformer area number and time range.

[0040] Batch computing module: A distributed computing framework built using Spark 3.4.1 is deployed, consisting of one master node and three slave nodes. The master node has 16 CPU cores and 32GB of memory, while the slave nodes have 24 CPU cores and 64GB of memory. Historical curve data stored in Hive is read via Spark SQL. This historical data is stored in monthly partitions and batch analysis tasks are executed, including monthly line loss rate trend analysis and statistics on abnormal line loss periods in transformer areas. The batch analysis results are synchronized to the ClickHouse cluster for real-time application access, with synchronization occurring daily at 2 AM.

[0041] like Figure 1As shown, this method follows an overall process of "data access - orchestration and calculation - dual data storage - application interaction" to achieve end-to-end automated processing of curve data. The implementation steps of this method are as follows:

[0042] Data Access: Smart meters and current transformers in the distribution area collect power data every minute. The data is then encapsulated into JSON format via the MQTT protocol and pushed to the `power_line_loss_data` topic in the Kafka cluster. The Kafka cluster uses a multi-partition and replication mechanism to ensure that data is not lost or duplicated, achieving highly reliable data transmission.

[0043] Data Stream Orchestration and Computation: Apache NiFi's KafkaConsumer processor monitors Kafka topics in real time, pulls data in batches, and transmits it to the CalculateRecord processor. This processor first generates minute-level indicator curves based on the pulled time-series data, extracting the voltage, current, total active power, and cumulative energy values ​​corresponding to each minute's data point; forming a single-meter-level "time-cumulative energy" indicator curve; simultaneously, it aggregates the cumulative energy values ​​of all single meters within the same distribution area to generate a distribution area-level "time-cumulative total energy" indicator curve; furthermore, for distribution area line loss scenarios, it provides three methods for obtaining power supply and sales volume—Method one involves obtaining the distribution area-level "time-cumulative total active power" indicator curve and the cumulative energy values ​​of all user meters within the distribution area. The "Time-Total Active Power" indicator curve is integrated over a 1-minute interval; Method 2 extracts the difference between the beginning and end of the corresponding time period from the "Time-Cumulative Total Energy" indicator curve at the transformer substation level and the "Time-Cumulative Energy" indicator curve at the user side; Method 3 directly reads the real-time recorded value of the "Cumulative Energy Increment" field in the cumulative energy indicator curve. All three methods can obtain the power supply and sales volume of the transformer substation during that time period. Finally, these values ​​are substituted into the line loss rate calculation formula (line loss rate = (power supply - sales volume) / power supply × 100%) to obtain the real-time line loss rate. The feature extraction stage simultaneously extracts key features of the energy indicator curve, including the cumulative energy increment, curve abrupt change points, and time period averages, providing data support for the judgment of abnormal electrical parameters. Figure 2 As shown, the data flow orchestration and computation process includes key steps such as monitoring, aggregation computation, feature extraction, and process scheduling to ensure the real-time performance and accuracy of the computation process. The process scheduling unit uses a load balancing mechanism to prioritize the allocation of line loss rate calculation tasks based on the power indicator curve to slave nodes, avoiding single-node overload and ensuring real-time response of the computation process.

[0044] Dual Data Synchronous Storage: Apache NiFi, through the PutClickHouseRecord processor, synchronously writes the raw curve data and calculation results (including numerical statistics, line loss rate, cumulative energy increment, and aggregated cumulative total energy readings for the transformer area) along with a unified acquisition timestamp, transformer area number, and time interval identifier of the indicator curve into the corresponding data table of the ClickHouse cluster. ClickHouse optimizes the data storage structure through composite indexes, achieving co-location of raw data and calculation results within the same database, enabling data traceability without cross-system switching.

[0045] Data Application Interaction: The line loss monitoring dashboard application sends query requests to the ClickHouse cluster via HTTP protocol to obtain line loss rate data, raw power data, and single-meter-level "time-cumulative energy" indicator curve data and transformer area-level "time-cumulative total energy" indicator curve data for a specified transformer area and time range. It displays the transformer area line loss change curve and cumulative energy trend curve in real time. In addition to querying calculation results according to preset thresholds (line loss rate exceeding 10% or below -3%), the anomaly warning module has added an energy anomaly monitoring logic. When the minute-level cumulative energy increase exceeds the normal fluctuation range (such as a single increase exceeding 3 times the average increase of the same period in the transformer area), an early warning notification is triggered simultaneously. The data query interface allows staff to query historical data, calculation results, and raw and aggregated data of energy indicator curves by time, transformer area number, and other conditions, supporting the preparation of line loss analysis reports and the traceability of energy consumption.

[0046] Batch computation: Every day at 2:00 AM, the Spark framework starts batch computation tasks, reads the historical curve data of the previous day in Hive, and adds batch analysis logic based on the power consumption curve in addition to performing monthly line loss trend analysis and abnormal period statistics. This includes monthly power consumption trend analysis and power consumption increment abnormal period statistics. After obtaining the batch analysis results, the results are synchronized to the ClickHouse cluster through SparkClickHouseConnector to supplement the historical data analysis needs that cannot be covered by real-time computation, and at the same time provide data support for power load optimization and power consumption behavior analysis in the distribution area.

[0047] In summary, this embodiment uses power distribution area line loss monitoring as an application scenario. By clearly defining the specific configurations of modules for data access, data stream orchestration and computation, storage, application interaction, and batch computation, such as using Kafka for reliable data transmission, Apache NiFi for automated computation processes, ClickHouse for dual data storage, and Spark for historical data processing, it comprehensively presents a curve data processing solution based on data stream orchestration. The implementation steps clearly cover the entire data chain from acquisition, computation, storage to application, verifying the feasibility of the solution in real-time line loss calculation and historical data analysis scenarios. It also demonstrates its advantages in multi-module collaboration, data association storage, and scenario adaptability, fully proving that the technical solution of this invention can effectively solve the problems of fragmented and inefficient traditional curve data processing, and has practical application value.

[0048] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A curve calculation method based on data stream orchestration, characterized in that, The specific steps of this method are as follows: Step 1, Data Access: Receive curve data reported in real time from external sources through a distributed message bus. The curve data is time-series data with time as the horizontal axis and a specific physical quantity or index as the vertical axis. Step 2, Data Flow Orchestration and Calculation: By using a data flow automation tool to monitor the target data topic of the distributed message bus, the curve data is acquired in real time. The built-in aggregation calculation component of the data flow automation tool is used to aggregate and calculate the curve data according to preset business rules to obtain the calculation results. Step 3, Dual Data Synchronization Storage: Through the dual data synchronization mechanism of the data flow automation tool, the original curve data and calculation results are synchronously written into the columnar storage database for storage in the same database, carrying a unified time stamp and association identifier. Step 4, Data Application Interaction: The application service accesses the columnar storage database through standard network protocols to obtain raw curve data or calculation results to support business decisions and applications.

2. The curve calculation method based on data stream orchestration according to claim 1, characterized in that, The distributed message bus in step 1 is Kafka. The curve data includes voltage, current, and power time-series data in the power system or operating parameter time-series data of industrial equipment. Kafka is deployed in a cluster to transmit curve data.

3. The curve calculation method based on data stream orchestration according to claim 1, characterized in that, The data flow automation tool in step 2 is Apache NiFi. The aggregation calculation includes numerical statistical calculation and time series feature extraction. The numerical statistical calculation includes average value, peak value, valley value and summation. The time series feature extraction includes trend change rate and periodic analysis.

4. The curve calculation method based on data stream orchestration according to claim 1, characterized in that, In step 2, the data flow automation tool achieves real-time scheduling of the entire process through its built-in process scheduling component. The process scheduling component uses a priority queue to allocate data processing priorities and uses a dynamic load balancing mechanism to allocate computing resources to process curve data.

5. The curve calculation method based on data stream orchestration according to claim 1, characterized in that, The unified time stamp in step 3 is the collection timestamp of the curve data, and the association identifier includes the data source identifier; the columnar storage database is ClickHouse, which is deployed through a sharded cluster and indexed by the time stamp and association identifier to store the original curve data and calculation results.

6. The curve calculation method based on data stream orchestration according to claim 1, characterized in that, The standard network protocol in step 4 is the HTTP protocol. The application service directly initiates query requests to the columnar storage database through the HTTP protocol. The data application interaction also includes a microservice collaboration mechanism, that is, the request routing, user authentication and authorization and network attack protection are realized through the microservice gateway, and the automatic registration and node discovery of application services are realized through the microservice registry.

7. The curve calculation method based on data stream orchestration according to claim 1, characterized in that, The method further includes a batch computing step: reading historical curve data or periodic statistical data through a distributed computing framework, performing deep mining and batch analysis to obtain batch analysis results, and synchronizing the batch analysis results to a columnar storage database or a distributed data warehouse; the distributed computing framework is Spark, and the distributed data warehouse is Hive.

8. A curve calculation system based on data stream orchestration, the system being applicable to the curve calculation method based on data stream orchestration as described in any one of claims 1-7, characterized in that, The system includes: Data access module: Employs a distributed message bus to receive curve data reported in real time by external acquisition devices and transmit the curve data to the data stream orchestration and calculation module; the curve data is time-series data with time as the horizontal axis and a specific physical quantity or index as the vertical axis; Data Flow Orchestration and Computation Module: Employs automated data flow tools, including a data monitoring unit, an aggregation and computation unit, and a process scheduling unit. The data monitoring unit monitors the target data topics of the distributed message bus and acquires curve data in real time. The aggregation and computation unit uses the built-in aggregation and computation component to perform aggregation and computation on the curve data according to preset business rules to obtain the calculation results. The process scheduling unit schedules the data processing flow to achieve automated data flow from acquisition to computation. Dual data synchronization storage module: includes a columnar storage database for storing raw curve data and calculation results, and the raw curve data and calculation results carry a unified time stamp and association identifier; Data application interaction module: includes application service unit, which is used to access columnar storage database through standard network protocol to obtain raw curve data or calculation results to support business decision-making and application display.

Images