Automatic index calculation method based on information model and real-time data driving

Abstract

The invention provides an automatic index calculation method based on an information model and real-time data driving, which relates to the technical field of data driving and comprises the steps of receiving multi-source heterogeneous data and establishing an information model, constructing an index dependency relation directed acyclic graph, establishing an incremental calculation engine and configuring a calculation rule and a triggering mode. And data driving and timing triggering calculation are supported, and real-time automatic calculation of indexes is realized. According to the method, the index calculation efficiency can be improved, the calculation resource consumption is reduced, the data real-time performance is ensured, and the system expandability and the index management capability are improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to an automatic indicator calculation method based on information models and real-time data driving. Background Art

[0002] Currently, in the industrial field, the real-time production data generated by massive multi-source heterogeneous terminal devices is showing an exponential growth trend. The current industrial data application has the following technical bottlenecks: (1) Multi-dimensional service defects: Due to the lack of a unified semantic abstraction layer for device native data, different business departments need to repeatedly build data interpretation models in scenarios such as energy consumption analysis, equipment operation and maintenance, and production scheduling, resulting in low efficiency in cross-domain data collaboration; (2) Excessive system architecture coupling: Under the traditional chimney-style development model, each application system directly connects to the original data source and needs to independently implement the full-process processing logic including data cleaning, indicator calculation, and dimension mapping. When business rules change, the entire link code needs to be reconstructed; (3) Waste of computing resources: Cascade computing tasks under the batch processing architecture are prone to generate redundant data copies. There is a cumulative effect of computing delay when dealing with time-series data stream processing, resulting in a super-linear growth relationship between system response time and data size.

[0003] In existing technologies, although solutions such as ETL can achieve basic data conversion, they have two inherent defects: first, computing task scheduling is based on a fixed time window and cannot respond to real-time changes in data sources; second, dimension aggregation logic is hard-coded in the application layer, resulting in extremely low indicator reuse rate; and other streaming computing frameworks, although they improve timeliness, lack the ability to dynamically model indicator hierarchical dependencies, resulting in a high degree of invalid calculations in complex business scenarios. Summary of the Invention

[0004] The embodiment of the present invention provides an automatic indicator calculation method based on an information model and real-time data drive, which can solve the problems in the prior art.

[0005] A first aspect of an embodiment of the present invention provides an automated indicator calculation method based on an information model and driven by real-time data, comprising: Receiving multi-source heterogeneous data, establishing an information model, and mapping the multi-source heterogeneous data to the information model, wherein the information model is used to classify and manage indicators, including remote indicators, input indicators, and calculated indicators; Constructing a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine hierarchical dependencies between different indicators, and topologically sorting the indicators based on the hierarchical dependencies; Based on the topological sorting results, an incremental calculation engine is established. When a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators; Configuring calculation rules for the calculation indicators, wherein the calculation rules include at least one of arithmetic formulas, system functions, and database query statements; Configure the calculation trigger mode of the calculation indicator, which includes data-driven trigger calculation and timed trigger calculation; When data-driven triggered calculations are used, the incremental calculation engine identifies data changes in underlying indicators and automatically triggers calculations of associated upper-level indicators based on the directed acyclic graph; When timing trigger calculation is adopted, the value of the current indicator is calculated according to the preset time period, and the calculation instruction is sent to the incremental calculation engine through the distributed message queue to trigger the calculation of the upper-level indicator; The calculation results are written into a distributed database for storage, and an indicator data query interface is provided to external applications.

[0006] The multi-source heterogeneous data includes data collected by industrial Internet of Things terminals, manually entered data, and database stored data; Based on the topological sorting result, an incremental calculation engine is established. When a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators, including: Constructing an incremental computing engine, which includes a data change monitoring module, an impact range analysis module, and a computing task scheduling module; Subscribe to the data change message of the underlying indicator through the data change monitoring module to obtain the identification information and change data of the changed indicator; Inputting the identification information of the change indicator into the impact range analysis module, tracing upward based on the topological sorting sequence, identifying all upper-level indicators that are directly and indirectly dependent on the change indicator, and generating a set of indicators to be calculated; Prioritizing the indicators in the set of indicators to be calculated according to the topological sorting sequence, wherein the priority is positively correlated with the level of the indicator in the topological sorting sequence; Creating computing tasks according to the priority order by the computing task scheduling module, and organizing indicators with the same priority into computing task batches; When executing each batch of computing tasks, multiple indicators within the same batch are calculated in parallel, and computing tasks of different batches are executed serially; The calculation status and results of each indicator are recorded. If an exception occurs during the calculation process, the exception information is associated with the corresponding indicator and the calculation of its upper-level indicator is terminated.

[0007] When timing trigger calculation is used, the value of the current indicator is calculated according to the preset time period, and calculation instructions are sent to the incremental calculation engine through the distributed message queue. Triggering the calculation of the upper-level indicator includes: Obtaining timing configuration information for calculating an indicator from an indicator configuration database, wherein the timing configuration information includes an indicator identifier, a calculation period, and a calculation time point; Construct a scheduled task scheduler to generate a CRON expression based on the timing configuration information, wherein the CRON expression is used to describe the time rule for indicator calculation; Organize the indicator identifier and the CRON expression into a scheduled computing task and register it to a distributed scheduling center; The distributed scheduling center determines the calculation triggering timing according to the CRON expression, and when the calculation time point is reached, triggers the calculation of the current timing indicator and generates a calculation triggering message.

[0008] The method further comprises: Writing the calculation trigger message into a distributed message queue, wherein the calculation trigger message includes identification information of the indicator to be calculated and a calculation time range; The incremental calculation engine monitors the distributed message queue and obtains the calculation trigger message; The incremental calculation engine extracts the original data of the corresponding time period from the data source according to the calculation time range in the calculation trigger message; Calculate the index based on the original data and write the calculation results into the real-time database and the historical database respectively; Record the execution status and completion time of the computing task, and return the execution result to the distributed scheduling center after the calculation is completed.

[0009] A second aspect of an embodiment of the present invention provides an automated indicator calculation system based on an information model and driven by real-time data, including: The first unit is configured to receive multi-source heterogeneous data, establish an information model, and map the multi-source heterogeneous data to the information model, wherein the information model is used to classify and manage indicators, including remote indicators, input indicators, and calculated indicators; The second unit is used to construct a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine the hierarchical dependencies between different indicators and to topologically sort the indicators based on the hierarchical dependencies; The third unit is configured to establish an incremental calculation engine based on the topological sorting result, and when a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators; A fourth unit is configured to configure a calculation rule for the calculation indicator, wherein the calculation rule includes at least one of arithmetic operation formulas, system functions, and database query statements; A fifth unit is used to configure a calculation triggering mode of the calculation indicator, wherein the calculation triggering mode includes data-driven triggering calculation and timed triggering calculation; A sixth unit is configured to, when data-driven triggered calculation is adopted, identify data changes of underlying indicators through the incremental calculation engine and automatically trigger calculation of associated upper-level indicators according to the directed acyclic graph; The seventh unit is used to calculate the value of the current indicator according to a preset time period when timing trigger calculation is adopted, and send a calculation instruction to the incremental calculation engine through a distributed message queue to trigger the calculation of the upper-level indicator; The eighth unit is used to write the calculation results into a distributed database for storage and provide an indicator data query interface to external applications.

[0010] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0011] According to a fourth aspect of the embodiments of the present invention, A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0012] The beneficial effects of this application are as follows: By establishing an information model to uniformly manage multi-source heterogeneous data and constructing a directed acyclic graph of indicator dependencies, the system can clearly identify the dependencies between indicators, achieve efficient management and calculation of indicators, and reduce the difficulty of indicator maintenance in complex business scenarios.

[0013] By adopting an incremental computing engine and a data-driven trigger mechanism, when the underlying indicator data changes, the system can intelligently identify and recalculate only the affected upper-level indicators, avoiding the waste of resources caused by full calculations and improving computing efficiency and system response speed.

[0014] By flexibly configuring calculation rules and triggering methods, combined with distributed message queue and distributed database technologies, the automation and scalability of the indicator calculation process are achieved, meeting the real-time computing needs in different business scenarios and improving the timeliness and accuracy of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flow chart of an automatic indicator calculation method based on an information model and real-time data driven according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0017] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0018] Figure 1 This is a flow chart of an automatic indicator calculation method based on an information model and real-time data driven according to an embodiment of the present invention. Figure 1 As shown, the method includes: Receiving multi-source heterogeneous data, establishing an information model, and mapping the multi-source heterogeneous data to the information model, wherein the information model is used to classify and manage indicators, including remote indicators, input indicators, and calculated indicators; Constructing a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine hierarchical dependencies between different indicators, and topologically sorting the indicators based on the hierarchical dependencies; Based on the topological sorting results, an incremental calculation engine is established. When a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators; Configuring calculation rules for the calculation indicators, wherein the calculation rules include at least one of arithmetic formulas, system functions, and database query statements; Configure the calculation trigger mode of the calculation indicator, which includes data-driven trigger calculation and timed trigger calculation; When data-driven triggered calculations are used, the incremental calculation engine identifies data changes in underlying indicators and automatically triggers calculations of associated upper-level indicators based on the directed acyclic graph; When timing trigger calculation is adopted, the value of the current indicator is calculated according to the preset time period, and the calculation instruction is sent to the incremental calculation engine through the distributed message queue to trigger the calculation of the upper-level indicator; The calculation results are written into a distributed database for storage, and an indicator data query interface is provided to external applications.

[0019] In an optional embodiment, the multi-source heterogeneous data includes data collected by industrial Internet of Things terminals, manually entered data, and database stored data; To collect data from industrial IoT terminals, the system deploys multiple data collection nodes, distributed near key factory equipment. Each node is equipped with a sensor module for real-time monitoring of equipment operating parameters such as temperature, pressure, and vibration frequency. The collected data is encrypted via a wireless transmission module and sent to an edge computing server. For example, a temperature sensor on a blast furnace in a steel plant collects temperature data every 5 seconds. The data format is: {Device ID: "HF-01", Parameter Type: "Temperature", Value: 1350.5, Unit: "°C", Timestamp: "2023-10-15 09:30:25"}.

[0020] For manual data entry, the system provides a user-friendly interface, allowing operators to enter unstructured information such as equipment inspection records and fault descriptions via a touchscreen or mobile device. To ensure data quality, the interface incorporates data validation mechanisms, including value range checks and logical relationship verification. For example, the data structure of an equipment inspection record might be: {Inspector: "Mr. Zhang", Equipment ID: "HF-01", Inspection Item: "Bearing Status", Status Description: "Normal", Note: "Slight noise but within the acceptable range", Inspection Time: "2023-10-15 10:15:00"}.

[0021] The database stores data such as historical production records, equipment archives, and maintenance records. This data is stored in a relational database and accessed using structured query language (SQL). The system regularly extracts relevant data from the enterprise resource planning (ERP) and manufacturing execution system (MES) and converts it into a unified format. For example, the structure of an equipment maintenance record is: {Maintenance Order Number: "R20231012-005", Equipment ID: "HF-01", Fault Type: "Temperature Sensor Fault", Repair Details: "Replace Sensor Element", Repair Personnel: "Li Xiuxiu", Start Time: "2023-10-12 08:30:00", Completion Time: "2023-10-12 10:45:00", Repair Cost: 3500.00}.

[0022] The data processing process begins with data cleansing, employing different cleaning strategies for different data types. For data collected by industrial IoT terminals, the system checks whether the values ​​are within a reasonable range. For example, blast furnace temperatures typically range from 1200°C to 1500°C. Values ​​outside this range are flagged as anomalies. For continuous time series data, the system uses a sliding window mean method to fill missing values, with a window size of five sampling points before and after. For obvious data errors, such as a sudden temperature jump from 1350°C to 135°C (possibly due to a decimal point error), the system automatically corrects them based on the context.

[0023] For manually entered data, the system checks that required fields are complete and that the text contains no sensitive words or meaningless content. For example, if words like "test" or "placeholder" appear in the notes field, the system prompts the operator to modify them. For descriptive text, the system extracts key information and standardizes it. For example, descriptions like "bearing is a bit noisy" and "bearing is noisy" are uniformly categorized as "bearing abnormal noise."

[0024] For data stored in the database, the system checks data format consistency, ensuring that time formats, units of measurement, and other metrics conform to standardized standards. For example, date and time formats across different systems are uniformly converted to the "YYYY-MM-DD HH:MM:SS" format. For inconsistent units in historical data, such as some records using Celsius and others using Fahrenheit, the system automatically converts units to ensure that all temperature data is in Celsius.

[0025] After data cleaning is complete, the system performs data integration and fusion. First, a unified data model is established to define the relationships between various data types. Core entities include devices, parameters, events, and people, and these entities are connected through relationships. For example, a device entity contains attributes such as device identification, type, and installation location; a parameter entity contains attributes such as parameter name, value, and unit; an event entity contains attributes such as event type, occurrence time, and severity; and a person entity contains attributes such as name, role, and contact information.

[0026] During the data fusion phase, the system performs time alignment and cross-validation on data from different sources for the same equipment. For example, if a blast furnace device has a temperature data of 1350.5°C collected by a sensor at the same time (2023-10-15 09:30:00), and an operator records an observation of approximately 1345°C, the system compares the two values. If the difference is within the allowable range (±10°C), the sensor data is used as the correct value. If the difference is too large, the data is marked as pending verification.

[0027] To integrate multidimensional data, the system constructs an equipment health index model that comprehensively considers real-time equipment parameters, historical fault records, and maintenance status. Taking blast furnace equipment as an example, the health index calculation takes into account factors such as temperature stability (the deviation between the current temperature and the average temperature over the past 24 hours), recent fault frequency (number of faults in the past 30 days), and the replacement time of key components (number of days since the last overhaul) to generate a health score between 0 and 100. For example, a blast furnace's current health index is 85, indicating that the equipment is in good condition.

[0028] The final stage of data processing involves knowledge extraction and decision support. Based on the processed data, the system constructs a database of equipment operating patterns, recording normal operating parameter ranges and typical fault characteristics. By comparing real-time data with the database, the system can identify potential problems early. For example, if a blast furnace's temperature fluctuations exceed ±30°C for three consecutive hours, and the oxygen flow rate is unstable, the system generates an alert: {Equipment ID: "HF-01", Alert Type: "Temperature Unstable", Severity: "Medium", Possible Cause: "Oxygen Supply System Fluctuation", Recommended Action: "Check Oxygen Control Valve", Alert Time: "2023-10-15 15:20:00"}.

[0029] Through the above-mentioned multi-source heterogeneous data processing method, industrial enterprises can make full use of various data resources to achieve real-time monitoring of equipment status, fault prediction and maintenance decision optimization, thereby improving production efficiency and equipment reliability.

[0030] In an optional embodiment, an incremental calculation engine is established based on the topological sorting result. When a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators, including: Constructing an incremental computing engine, which includes a data change monitoring module, an impact range analysis module, and a computing task scheduling module; Subscribe to the data change message of the underlying indicator through the data change monitoring module to obtain the identification information and change data of the changed indicator; Inputting the identification information of the change indicator into the impact range analysis module, tracing upward based on the topological sorting sequence, identifying all upper-level indicators that are directly and indirectly dependent on the change indicator, and generating a set of indicators to be calculated; Prioritizing the indicators in the set of indicators to be calculated according to the topological sorting sequence, wherein the priority is positively correlated with the level of the indicator in the topological sorting sequence; Creating computing tasks according to the priority order by the computing task scheduling module, and organizing indicators with the same priority into computing task batches; When executing each batch of computing tasks, multiple indicators within the same batch are calculated in parallel, and computing tasks of different batches are executed serially; The calculation status and results of each indicator are recorded. If an exception occurs during the calculation process, the exception information is associated with the corresponding indicator and the calculation of its upper-level indicator is terminated.

[0031] Build an incremental computing engine, which includes a data change monitoring module, an impact analysis module, and a computing task scheduling module. The data change monitoring module is responsible for subscribing to data change messages for underlying indicators; the impact analysis module is responsible for analyzing the impact of changed indicators on upper-level indicators; and the computing task scheduling module is responsible for creating and scheduling computing tasks.

[0032] The data change monitoring module subscribes to data change notifications for underlying metrics through a message queue mechanism. For example, a Kafka message queue can be used to create a topic for each underlying metric. When metric data changes, the data source system publishes a change notification to the corresponding topic. The data change monitoring module acts as a consumer, subscribing to these topics and receiving change notifications in real time. Change notifications contain the identifier of the changed metric (such as the metric ID and name) and the changed data (such as the new value and the change timestamp).

[0033] In the specific implementation, the data change monitoring module can use the Spring Kafka framework to configure consumers. The code example is as follows: After receiving the change message, the data change monitoring module extracts the identification information of the changed indicator (such as indicator ID: "METRIC_001") and the changed data (such as new value: 123.45, change time: 2023-05-20 10:15:30), and passes the identification information to the impact range analysis module.

[0034] The impact analysis module uses a pre-calculated topological sorting sequence to trace back and identify all upper-level indicators that directly and indirectly depend on the changed indicator. For example, suppose there is an indicator dependency relationship: indicator A depends on indicators B and C, indicator B depends on indicator D, and indicator C depends on indicator E. The topological sorting sequence is [D, E, B, C, A]. If indicator D changes, the impact analysis module will identify that indicators B and A are affected and generate the set of indicators to be calculated {B, A}.

[0035] The impact analysis module uses a depth-first search algorithm to analyze the impact scope. Based on the indicator dependency graph, starting from the changed indicator, the module traverses upward along the dependency edges, adding all reachable indicators to the set of indicators to be calculated. For example, when the indicator "METRIC_001" (sales) changes, the system identifies the directly dependent indicators "METRIC_005" (gross profit margin) and "METRIC_007" (sales growth rate), as well as the indirectly dependent indicator "METRIC_012" (comprehensive performance index), forming the set of indicators to be calculated: {METRIC_005, METRIC_007, METRIC_012}.

[0036] After the impact analysis is complete, the system prioritizes the metrics in the set of metrics to be calculated based on the topological sorting sequence. Priority is positively correlated with the rank of the metric in the topological sorting sequence; lower ranks have higher priorities. For the set of metrics to be calculated {B, A} in the example above, since B's rank is lower than A's in the topological sorting sequence [D, E, B, C, A], B has a higher priority than A, resulting in the sorted result of [B, A].

[0037] In a practical scenario, for example, if the set of metrics to be calculated is {METRIC_005, METRIC_007, METRIC_012}, the priority order obtained according to the topological sorting sequence may be [METRIC_005, METRIC_007, METRIC_012], which means that METRIC_005 and METRIC_007 should be calculated first, and then METRIC_012.

[0038] The computational task scheduling module creates computational tasks in order of priority and organizes metrics of the same priority into computational task batches. For the above example, the possible computational task batches are: the first batch {METRIC_005, METRIC_007}, and the second batch {METRIC_012}.

[0039] When executing calculation tasks, the system calculates multiple metrics within the same batch in parallel and executes calculation tasks for different batches serially. For example, for the first batch {METRIC_005, METRIC_007}, the system creates two parallel calculation threads to calculate METRIC_005 and METRIC_007 respectively. After the first batch is completed, the second batch {METRIC_012} is calculated.

[0040] In practice, a ThreadPoolExecutor thread pool is used to manage parallel computation tasks, creating a CountDownLatch for each batch to ensure serial execution between batches. Each computation task retrieves the required dependency metric data from a database or cache, executes the computation logic, and updates the computation results back to the data storage system.

[0041] The system records the calculation status (e.g., "Waiting for Calculation," "Calculating," "Completed," "Failed") and results (e.g., calculated value, calculation time) of each metric. For exceptions that occur during calculation, the system associates the exception information (e.g., exception type, message, and stack trace) with the corresponding metric, marks it as "Failed," and terminates the calculation of the upper-level metric.

[0042] For example, if the calculation of METRIC_005 fails, the system will log the exception message: "Division by zero error, time: 2023-05-20 10:16:05" and mark the status of METRIC_005 as "Calculation failed." Because METRIC_012 depends on METRIC_005, the system will cancel the calculation task for METRIC_012 and mark its status as "Cancelled, dependent metric calculation failed."

[0043] Through the above method, the incremental calculation engine can accurately identify the affected upper-level indicators when the underlying indicator data changes, and efficiently recalculate them in a reasonable order, avoiding unnecessary calculations of unaffected indicators and significantly improving system performance and resource utilization.

[0044] In an optional embodiment, when timing-triggered calculation is adopted, the value of the current indicator is calculated according to a preset time period, and a calculation instruction is sent to the incremental calculation engine through a distributed message queue. Triggering the calculation of the upper-level indicator includes: Obtaining timing configuration information for calculating an indicator from an indicator configuration database, wherein the timing configuration information includes an indicator identifier, a calculation period, and a calculation time point; Construct a scheduled task scheduler to generate a CRON expression based on the timing configuration information, wherein the CRON expression is used to describe the time rule for indicator calculation; Organize the indicator identifier and the CRON expression into a scheduled computing task and register it to a distributed scheduling center; The distributed scheduling center determines the calculation triggering timing according to the CRON expression, and when the calculation time point is reached, triggers the calculation of the current timing indicator and generates a calculation triggering message.

[0045] In a specific embodiment, the method includes the steps of obtaining timing configuration information from an indicator configuration database, building a timing task scheduler, generating and registering a timing calculation task, determining a triggering timing and generating a calculation trigger message.

[0046] First, the system needs to obtain the timing configuration information for calculating the indicator from the indicator configuration database. This configuration information includes at least three items: the indicator identifier, the calculation period, and the calculation time point. For example, for the daily active users (DAU) indicator on an e-commerce platform, the configuration information might include the indicator identifier "DAU_DAILY", the calculation period "Daily", and the calculation time point "2:00 AM the next day". The system extracts this information from the indicator configuration table using a SQL query statement, such as "SELECT indicator_id, calc_cycle, calc_time FROM indicator_config WHERE indicator_type = 'timing'".

[0047] After obtaining the configuration information, the system builds a scheduled task scheduler. Based on the scheduled configuration information, the scheduler generates a CRON expression to describe the timing rules for metric calculations. A CRON expression consists of six fields: seconds, minutes, hours, days, months, and weeks. These fields can be combined to accurately describe the task execution time. For example, for the DAU metric above, if calculations are to be performed every day at 2:00 AM, the CRON expression can be expressed as "0 0 2 * * ?", where an asterisk indicates any value and a question mark indicates an unspecified value.

[0048] After receiving a task, the distributed scheduling center stores it in its task library and continuously determines whether the calculation trigger time has been reached based on the CRON expression. When the calculation time point is reached, the scheduling center generates a calculation trigger message containing information such as the indicator identifier, trigger time, and calculation parameters, and sends it to the message queue. When the calculation time point is reached, the scheduling center triggers the calculation of the scheduled indicator. After the calculation of the scheduled indicator is completed, the result of the scheduled calculation indicator is written to the distributed message queue. When the incremental calculation engine monitors the value of this indicator, it triggers the calculation of the upper-level indicator.

[0049] To ensure system reliability, the distributed dispatch center also needs to implement the following functions: 1. Task execution status monitoring: record the execution results of each trigger; 2. Failure retry mechanism: If the trigger message fails to be sent, retry according to the preset strategy; 3. Load balancing: Distributed locks and other mechanisms are used to ensure that the same task is not triggered repeatedly in a cluster environment; 4. Dynamic configuration update: Supports updating CRON expressions without restarting the system.

[0050] This method allows the system to accurately trigger metric calculations at specified times based on pre-set time rules, meeting business requirements for timely data calculations. This method is particularly suitable for business metrics that require regular updates, such as daily active users, weekly order conversion rates, and monthly sales.

[0051] In an optional embodiment, the method further includes: Writing the calculation trigger message into a distributed message queue, wherein the calculation trigger message includes identification information of the indicator to be calculated and a calculation time range; The incremental calculation engine monitors the distributed message queue and obtains the calculation trigger message; The incremental calculation engine extracts the original data of the corresponding time period from the data source according to the calculation time range in the calculation trigger message; Calculate the index based on the original data and write the calculation results into the real-time database and the historical database respectively; Record the execution status and completion time of the computing task, and return the execution result to the distributed scheduling center after the calculation is completed.

[0052] First, during the system initialization phase, a distributed scheduling center is established. This scheduling center uses a ZooKeeper-based distributed coordination service to manage task scheduling, monitoring, and exception handling. The distributed scheduling center is configured with multiple compute nodes, each with independent computing resources and fault recovery mechanisms. Furthermore, a distributed message queue based on Kafka is established to store and transmit computation trigger messages.

[0053] After a calculation trigger message is written to the Kafka message queue, the system assigns it a specific partition and offset to ensure accurate tracking and processing. For example, the message might be assigned to partition 2 of the "indicator_calculation" topic at offset 1024.

[0054] Next, the incremental computing engine monitors the distributed message queue through a consumer group mechanism. Implemented based on Flink or Spark Streaming, the incremental computing engine continuously consumes new messages from the message queue. When a new computation trigger message is detected, the incremental computing engine parses the message content and extracts information such as the metric identifier and computation time range.

[0055] The incremental calculation engine extracts raw data for the corresponding time period from the data source based on the parsed calculation time range. The data source can be a distributed file system HDFS, a relational database MySQL, or a NoSQL database MongoDB.

[0056] After acquiring the raw data, the incremental calculation engine performs indicator calculations based on predefined calculation logic. This calculation logic typically includes operations such as data cleaning, aggregation, and statistics. For the daily sales indicator, the calculation engine aggregates sales records by different dimensions. The specific steps include: 1. Data cleaning: Filter out invalid records, such as those with negative or abnormally large amounts.

[0057] 2. Data conversion: Convert sales time into a standard time format.

[0058] 3. Data aggregation: Group by product ID and store ID, and calculate the total sales amount, sales quantity, etc. for each group.

[0059] After the calculation is complete, the results are written to two data storage systems: a real-time database and a historical database. The real-time database uses a Redis cluster with master-slave replication and persistence to support high-concurrency real-time queries. The historical database uses ClickHouse columnar storage to optimize data compression and aggregate query performance.

[0060] Throughout the computation process, the system records the execution status and key time points of computation tasks. Execution statuses include PENDING, RUNNING, COMPLETED, and FAILED. This status information is stored in the task status table in the distributed scheduling center and can be used for task tracking and exception handling.

[0061] After receiving the execution results, the distributed scheduling center will update the task status and trigger subsequent processes according to the configuration, such as data visualization updates, exception alarms, or downstream task scheduling.

[0062] Through the above detailed implementation steps, this method realizes efficient and reliable incremental indicator calculation based on a distributed architecture, which can meet the data processing needs of enterprise-level applications.

[0063] The embodiment of the present invention is based on an information model and a real-time data-driven automatic indicator calculation system, including: The first unit is configured to receive multi-source heterogeneous data, establish an information model, and map the multi-source heterogeneous data to the information model, wherein the information model is used to classify and manage indicators, including remote indicators, input indicators, and calculated indicators; The second unit is used to construct a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine the hierarchical dependencies between different indicators and to topologically sort the indicators based on the hierarchical dependencies; The third unit is configured to establish an incremental calculation engine based on the topological sorting result, and when a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators; A fourth unit is configured to configure a calculation rule for the calculation indicator, wherein the calculation rule includes at least one of arithmetic operation formulas, system functions, and database query statements; A fifth unit is used to configure a calculation triggering mode of the calculation indicator, wherein the calculation triggering mode includes data-driven triggering calculation and timed triggering calculation; A sixth unit is configured to, when data-driven triggered calculation is adopted, identify data changes of underlying indicators through the incremental calculation engine and automatically trigger calculation of associated upper-level indicators according to the directed acyclic graph; The seventh unit is used to calculate the value of the current indicator according to a preset time period when timing trigger calculation is adopted, and send a calculation instruction to the incremental calculation engine through a distributed message queue to trigger the calculation of the upper-level indicator; The eighth unit is used to write the calculation results into a distributed database for storage and provide an indicator data query interface to external applications.

[0064] According to a third aspect of an embodiment of the present invention, an electronic device is provided, including: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0065] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described above is implemented.

[0066] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated indicator calculation method based on information models and real-time data driven, characterized by: include: Receiving multi-source heterogeneous data, establishing an information model, and mapping the multi-source heterogeneous data to the information model, wherein the information model is used to classify and manage indicators, including remote indicators, input indicators, and calculated indicators; Constructing a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine hierarchical dependencies between different indicators, and topologically sorting the indicators based on the hierarchical dependencies; Based on the topological sorting results, an incremental calculation engine is established. When a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators; Configuring calculation rules for the calculation indicators, wherein the calculation rules include at least one of arithmetic formulas, system functions, and database query statements; Configure the calculation trigger mode of the calculation indicator, which includes data-driven trigger calculation and timed trigger calculation; When data-driven triggered calculations are used, the incremental calculation engine identifies data changes in underlying indicators and automatically triggers calculations of associated upper-level indicators based on the directed acyclic graph; When timing trigger calculation is adopted, the value of the current indicator is calculated according to the preset time period, and the calculation instruction is sent to the incremental calculation engine through the distributed message queue to trigger the calculation of the upper-level indicator; The calculation results are written into a distributed database for storage, and an indicator data query interface is provided to external applications.

2. The method according to claim 1, characterized in that The multi-source heterogeneous data includes data collected by industrial Internet of Things terminals, manually entered data and database stored data.

3. The method according to claim 1, characterized in that Constructing a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine hierarchical dependencies between different indicators, and topologically sorting the indicators based on the hierarchical dependencies includes: Obtain identification information of multiple indicators and corresponding calculation rules, wherein the calculation rules include reference relationships between indicators; Analyze the input indicator and output indicator of each indicator according to the calculation rule, use the input indicator as the starting point of the directed edge, and use the output indicator as the end point of the directed edge; Establishing an indicator node set, and adding identification information of the multiple indicators as graph nodes to the indicator node set; Establishing an indicator edge set, establishing directed edges between corresponding graph nodes according to the reference relationship between the input indicator and the output indicator, and adding the directed edges to the indicator edge set; Constructing a directed graph based on the indicator node set and the indicator edge set, and performing loop detection on the directed graph; If a loop is detected, loop prompt information is returned to the user; if no loop is detected, the indicators are hierarchical based on the in-degree values ​​of the nodes in the directed graph to obtain a hierarchical dependency relationship of the indicators; According to the hierarchical dependency, a depth-first search algorithm is used to traverse the nodes in the directed graph to generate a topologically sorted sequence of indicators.

4. The method according to claim 1, wherein Based on the topological sorting result, an incremental calculation engine is established. When a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators, including: Constructing an incremental computing engine, which includes a data change monitoring module, an impact range analysis module, and a computing task scheduling module; Subscribe to the data change message of the underlying indicator through the data change monitoring module to obtain the identification information and change data of the changed indicator; Inputting the identification information of the change indicator into the impact range analysis module, tracing upward based on the topological sorting sequence, identifying all upper-level indicators that are directly and indirectly dependent on the change indicator, and generating a set of indicators to be calculated; Prioritizing the indicators in the set of indicators to be calculated according to the topological sorting sequence, wherein the priority is positively correlated with the level of the indicator in the topological sorting sequence; Creating computing tasks according to the priority order by the computing task scheduling module, and organizing indicators with the same priority into computing task batches; When executing each batch of computing tasks, multiple indicators within the same batch are calculated in parallel, and computing tasks of different batches are executed serially; The calculation status and results of each indicator are recorded. If an exception occurs during the calculation process, the exception information is associated with the corresponding indicator and the calculation of its upper-level indicator is terminated.

5. The method according to claim 1, wherein When timing trigger calculation is used, the value of the current indicator is calculated according to the preset time period, and calculation instructions are sent to the incremental calculation engine through the distributed message queue. Triggering the calculation of the upper-level indicator includes: Obtaining timing configuration information for calculating an indicator from an indicator configuration database, wherein the timing configuration information includes an indicator identifier, a calculation period, and a calculation time point; Construct a scheduled task scheduler to generate a CRON expression based on the timing configuration information, wherein the CRON expression is used to describe the time rule for indicator calculation; Organize the indicator identifier and the CRON expression into a scheduled computing task and register it to a distributed scheduling center; The distributed scheduling center determines the calculation triggering timing according to the CRON expression, and when the calculation time point is reached, triggers the calculation of the current timing indicator and generates a calculation triggering message.

6. The method according to claim 5, characterized in that The method further comprises: Writing the calculation trigger message into a distributed message queue, wherein the calculation trigger message includes identification information of the indicator to be calculated and a calculation time range; The incremental calculation engine monitors the distributed message queue and obtains the calculation trigger message; The incremental calculation engine extracts the original data of the corresponding time period from the data source according to the calculation time range in the calculation trigger message; Calculate the index based on the original data and write the calculation results into the real-time database and the historical database respectively; Record the execution status and completion time of the computing task, and return the execution result to the distributed scheduling center after the calculation is completed.

7. An automated indicator calculation system based on an information model and real-time data driven, for implementing the method according to any one of claims 1 to 6, characterized in that: include: The first unit is configured to receive multi-source heterogeneous data, establish an information model, and map the multi-source heterogeneous data to the information model, wherein the information model is used to classify and manage indicators, including remote indicators, input indicators, and calculated indicators; The second unit is used to construct a directed acyclic graph of indicator dependencies, wherein the directed acyclic graph is used to determine the hierarchical dependencies between different indicators and to topologically sort the indicators based on the hierarchical dependencies; The third unit is configured to establish an incremental calculation engine based on the topological sorting result, and when a change is detected in the underlying indicator data, the incremental calculation engine only recalculates the affected upper-level indicators; A fourth unit is configured to configure a calculation rule for the calculation indicator, wherein the calculation rule includes at least one of arithmetic operation formulas, system functions, and database query statements; A fifth unit is used to configure a calculation triggering mode of the calculation indicator, wherein the calculation triggering mode includes data-driven triggering calculation and timed triggering calculation; A sixth unit is configured to, when data-driven triggered calculation is adopted, identify data changes of underlying indicators through the incremental calculation engine and automatically trigger calculation of associated upper-level indicators according to the directed acyclic graph; The seventh unit is used to calculate the value of the current indicator according to a preset time period when timing trigger calculation is adopted, and send a calculation instruction to the incremental calculation engine through a distributed message queue to trigger the calculation of the upper-level indicator; The eighth unit is used to write the calculation results into a distributed database for storage and provide an indicator data query interface to external applications.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images