Atomic index generation method based on real-time data stream processing

Abstract

The invention provides an atomic index generation method based on real-time data streaming processing, and relates to the technical field of data processing, and the method comprises the steps: generating a cache key based on an attribute unique code and an atomic index code, and querying whether a corresponding index calculation object exists in a cache; if the cache object does not exist, initializing an index calculation object comprising an index value, a start timestamp, a calculation type, an atomic index code and a calculation rule, and storing the index calculation object into a cache; if the cache object exists, corresponding index calculation operation is executed according to the calculation type, the index calculation operation comprises duration calculation, accumulated value calculation, aggregation calculation or user-defined rule calculation, and an index value in the cache is updated; writing the cached index values into a target database in batches according to a preset time period, and avoiding repeated writing by taking timestamps and the atomic index codes as unique identifiers; and resetting the index value in the cache to prepare for the next calculation period.

Description

Technical Field

[0001] This application belongs to the technical field of data processing, and particularly relates to a method for generating atomic metrics based on real-time data stream processing. Background Art

[0002] With the rapid development of the Internet of Things (IoT) and big data technologies, real-time data stream processing has become the core technology to support intelligent decision-making and system monitoring. As the basic measurement unit for describing the state of devices or systems, the real-time, accuracy, and flexibility of atomic metric calculation directly affect the efficiency of business analysis. However, when processing real-time data streams, the existing technologies have significant bottlenecks, especially in the generation of complex atomic metrics (such as duration, cumulative value, and custom rule calculation), and it is difficult to meet the requirements of high-concurrency and low-latency application scenarios.

[0003] Traditional big data platforms usually adopt ETL (Extract-Transform-Load) technology to perform offline calculations on historical data by periodically executing SQL aggregation functions (such as SUM, COUNT, etc.). This method requires frequent database operations. Especially when a large number of atomic metrics are calculated in parallel, it will cause a sudden increase in the database instantaneous load and affect the system stability. In addition, ETL cannot achieve real-time calculation and cannot process metrics that need to dynamically track the duration of the state (such as the device startup duration).

[0004] Although existing stream data processing engines (such as Apache Flink, Spark Streaming) support real-time data aggregation, their functions are mainly limited to simple numerical calculations (such as summation, mean). For atomic metrics that require state management (such as device state transition records) and complex rules (such as custom script calculation), the existing engines lack a flexible state caching mechanism and rule extension ability, resulting in limited application scope.

[0005] The application number is CN117668439A, a method and device for calculating streaming metrics of large-scale energy device IoT data, mentions improving the calculation efficiency through real-time data aggregation, but does not solve the problems of duration calculation and custom rule support, and the applicable scenarios are limited.

[0006] The application number is CN117764455A, a general metric management method and system for data, mentions optimizing the storage structure and query efficiency, but relies on non-real-time data processing and still has the defects of frequent database operations and inability to support dynamic duration calculation.

[0007] Therefore, the inventors found that existing solutions are difficult to support complex calculation types (such as duration tracking, cumulative value difference calculation) and custom rules (such as script-based dynamic logic). The lack of an efficient caching mechanism to record the timestamps and intermediate values of device state changes results in the inability to accurately generate persistent metrics (such as power-on duration). Traditional ETL and streaming engines frequently operate on databases, causing excessive instantaneous loads and affecting system stability. Summary of the Invention

[0008] This application provides a method for generating atomic metrics based on real-time data stream processing to solve the technical problems in the prior art that it is unable to efficiently support duration calculation, cumulative value difference calculation, and custom rule processing, and at the same time, the system performance degrades due to frequent database operations.

[0009] The technical solution adopted by this application is as follows:

[0010] An embodiment of this application provides a method for generating atomic metrics based on real-time data stream processing, including:

[0011] Generate a cache key based on the attribute unique encoding and the atomic metric encoding, and query whether there is a corresponding metric calculation object in the cache;

[0012] If there is no cache object, initialize a metric calculation object including a metric value, a start timestamp, a calculation type, an atomic metric encoding, and a calculation rule, and store it in the cache;

[0013] If there is the cache object, perform a corresponding metric calculation operation according to the calculation type. The metric calculation operation includes duration calculation, cumulative value calculation, aggregation calculation, or custom rule calculation, and update the metric value in the cache;

[0014] Batch write the metric values in the cache to the target database at a preset time period, and use the timestamp and the atomic metric encoding as the unique identifier to avoid duplicate writes;

[0015] Reset the metric values in the cache to prepare for the next calculation cycle.

[0016] According to an embodiment of this application, before generating a cache key based on the attribute unique encoding and the atomic metric encoding and querying whether there is a corresponding metric calculation object in the cache, it includes:

[0017] Real-time collect device data and perform formatting processing, and extract the data reporting time and the attribute unique encoding;

[0018] Subscribe to the data stream according to the atomic metric configuration rule and receive real-time data.

[0019] According to an embodiment of this application, the calculation type includes at least one of the following:

[0020] Aggregate calculations, including summation, average value, maximum value, or minimum value;

[0021] Duration calculation, generating the total duration by accumulating the difference in status change timestamps;

[0022] Accumulated value calculation, accumulating by the difference between the current value and the previous value;

[0023] Custom rule calculation, performing user-defined numerical operations based on a script engine.

[0024] According to an embodiment of the present application, the specific steps of the duration calculation include:

[0025] When it is detected that the device status changes from shutdown to startup, record the current timestamp as the start time;

[0026] When the device status remains startup, accumulate the difference between the current timestamp and the start time to the metric value;

[0027] When the status changes to shutdown, stop accumulating and record the final metric value.

[0028] According to an embodiment of the present application, the preset time period is at least one of whole minutes, whole hours, or zero o'clock.

[0029] According to an embodiment of the present application, when initializing the metric calculation object, if it is an accumulated value calculation, the initial metric value is set to 0, and the previous value is filled according to the received data.

[0030] According to an embodiment of the present application, the target database includes at least one of a relational database, a non-relational database, or a time series database.

[0031] According to an embodiment of the present application, subscribing to the data stream according to the atomic metric configuration rule and receiving real-time data is specifically:

[0032] Dynamically generate data subscription rules according to the atomic metric configuration, and receive data through a message queue or a real-time data stream.

[0033] A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the steps in the atomic metric generation method based on real-time data stream processing.

[0034] An electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor, and when the processor executes the program, it implements the steps in the atomic metric generation method based on real-time data stream processing.

[0035] Due to the adoption of the above technical solution, the beneficial effects obtained by the present application are:

[0036] This application generates a cache key based on the attribute unique encoding and the atomic index encoding, and queries whether there is a corresponding index calculation object in the cache: By using a unique encoding combination to generate the cache key, the system can quickly and accurately locate a specific index calculation object, greatly improving the query efficiency. This design reduces the workload of repeated calculations and ensures the consistency and accuracy of data processing.

[0037] If there is no cache object, initialize an index calculation object containing the index value, start timestamp, calculation type, atomic index encoding, and calculation rule, and store it in the cache: When a new index is encountered for the first time, the system can automatically initialize the corresponding calculation object and store it in the cache. This not only simplifies the system configuration process but also enhances its flexibility, enabling new indexes to be seamlessly integrated into the existing data processing flow without manual intervention.

[0038] If there is the cache object, perform the corresponding index calculation operation according to the calculation type. The index calculation operation includes duration calculation, cumulative value calculation, aggregation calculation, or custom rule calculation, and update the index value in the cache: According to different types of calculation requirements (such as duration, cumulative value, aggregation calculation, etc.), the system can accurately execute the corresponding calculation logic and update the index value in the cache in real time. This method ensures the accuracy and timeliness of data processing and meets the requirements of various complex business scenarios.

[0039] Batch write the index values in the cache to the target database at a preset time period, and use the timestamp and the atomic index encoding as the unique identifier to avoid duplicate writes: By adopting the batch write method, it is possible to reduce the pressure on the database while ensuring data real-time. In addition, using the timestamp and the atomic index encoding as the unique identifier effectively prevents the problem of duplicate writes, further enhancing the consistency and integrity of the data.

[0040] Reset the index values in the cache to prepare for the next calculation cycle: At the end of each calculation cycle, the system will reset the index values in the cache to prepare for data processing in the next cycle. This mechanism not only helps to maintain the continuity of data processing but also ensures the data independence within each cycle, facilitating subsequent analysis and auditing. Brief Description of the Drawings

[0041] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0042] Figure 1Schematic flowchart of an atomic metric generation method based on real-time data stream processing provided by an embodiment of the present application;

[0043] Figure 2 Computational flowchart of an atomic metric generation method based on real-time data stream processing provided by an embodiment of the present application;

[0044] Figure 3 Example diagram of the boot duration of a certain device in an atomic metric generation method based on real-time data stream processing provided by an embodiment of the present application. Detailed implementation manners

[0045] To more clearly illustrate the overall concept of the present application, the following will be described in detail by way of examples in conjunction with the accompanying drawings of the specification.

[0046] In the following description, many specific details are set forth in order to fully understand the present application. However, the present application may be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below. It should be noted that, without conflict, the embodiments of the present application and the features in each embodiment may be combined with each other.

[0047] In the present application, unless otherwise clearly defined and limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description of reference terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0048] Embodiment 1

[0049] As Figure 1 shown, an atomic metric generation method based on real-time data stream processing includes:

[0050] Generate a cache key based on the attribute unique encoding and the atomic metric encoding, and query whether there is a corresponding metric calculation object in the cache.

[0051] Specifically, the attribute unique encoding: This refers to the unique identifier of a specific attribute extracted from the data collected by the device. Each device or data source may have multiple different attributes (such as temperature, pressure, etc.), and each attribute has its unique encoding to distinguish it from other attributes.

[0052] Atomic Index Encoding: An atomic index refers to the basic measurement unit for describing the system state, such as the boot time, average temperature, etc. Each atomic index also has its unique encoding, which is used to distinguish different calculation logics and rules.

[0053] Combined Generation of Cache Key: By combining the "Unique Attribute Encoding" and the "Atomic Index Encoding" to form a composite key (for example, using string concatenation such as "Unique Attribute Encoding - Atomic Index Encoding"), this composite key is used as the key in the cache to quickly locate whether there is a calculation result for a certain atomic index under a specific attribute in the cache.

[0054] Checking the Cache: Once the cache key is generated, the system will use this key to query the local cache (it is mentioned in the text that the Caffeine tool is used for efficient cache management). If the value corresponding to the key exists in the cache, it means that the calculation has been performed for the same combination of attributes and atomic indices before, and the result is still valid.

[0055] If the corresponding index calculation object is not found in the cache, a new calculation object needs to be initialized. This object contains the initial index value, start timestamp, calculation type (aggregation, duration, custom rule), atomic index encoding, and possible calculation rules.

[0056] If the corresponding object is found in the cache, it means that part of the calculation has been completed or is in the process of calculation. At this time, the index value can be updated continuously according to the existing state, and subsequent operations can be carried out according to the specific calculation type (such as aggregation calculation, duration calculation, cumulative value calculation, or custom rule calculation).

[0057] For example, Device: A production device in a certain factory.

[0058] Unique Attribute Encoding: Assume it is DEVICE_001, representing a specific device.

[0059] Atomic Index Encoding: Assume it is MACHINE_ON_TIME, which is used to represent the boot time of the device.

[0060] Real-time Data Collection and Preliminary Processing:

[0061] Real-time collect the status data of the device (DEVICE_001) and perform formatting processing to ensure that the data carries the reporting time and the unique attribute encoding (DEVICE_001).

[0062] Generate Cache Key and Query Cache:

[0063] Generate a cache key using the combination of the attribute unique encoding (DEVICE_001) and the atomic metric encoding (MACHINE_ON_TIME), such as "DEVICE_001-MACHINE_ON_TIME".

[0064] The system uses this key to look up in the cache whether there is already a corresponding metric calculation object.

[0065] Check the cache result:

[0066] No cache case: If the system cannot find the cache object with the key "DEVICE_001-MACHINE_ON_TIME", initialize a new metric calculation object. This object contains information such as the initial machine-on time metric value (e.g., 0), start timestamp (current timestamp), calculation type (duration), atomic metric encoding (MACHINE_ON_TIME), etc., and write this object into the cache.

[0067] Cache hit case: If the cache object with the key "DEVICE_001-MACHINE_ON_TIME" is found, update the metric value in the cache object according to the currently received data. For example, if the previous value was in the shutdown state and the current value is in the startup state, update the start timestamp to the current timestamp and change the previous value to the startup state; conversely, if it changes from startup to shutdown, stop accumulating the duration and record the total machine-on time.

[0068] Periodically batch write to the database:

[0069] According to a preset time period (e.g., every hour), write the total machine-on time in the cache to the target database. The write operation must ensure that the timestamp + atomic metric encoding is used as the unique identifier to avoid duplication or confusion.

[0070] After the write is completed, reset the metric value in the cache to zero to prepare for the calculation in the next calculation cycle.

[0071] Furthermore, in addition to the basic device status (such as the boot time), more dimensions of data analysis can be introduced according to different business requirements, such as geographical location, environmental conditions, etc. This requires considering more attribute information when generating cache keys to support more detailed data segmentation and analysis. Dynamically adjust the cache policy: According to the actual running situation, dynamically adjust the cache expiration time or size limit. For some data that is frequently accessed but changes little, a longer cache validity period can be set; while for data that changes frequently, shorten the validity period to ensure data timeliness. Improve fault tolerance: Implement a data backup mechanism to ensure rapid recovery in case of system failures. For example, save the data to a temporary storage area before writing to the database, and if the write fails, retry from the temporary storage area. Optimize the rule engine: For the custom rule calculation part, the rule engine can be further optimized to support more complex logical judgments and processing, while reducing calculation latency. For example, introduce a machine learning model for predictive maintenance. Distributed cache: Use a distributed cache solution (such as Redis) instead of a local cache to improve the scalability and reliability of the system. Especially when dealing with large-scale data streams, distributed caches can provide better performance and higher availability. Event-driven architecture: Combine an event-driven architecture to trigger corresponding processing flows when new data arrives, rather than polling regularly. This approach can utilize resources more efficiently and respond promptly to data changes. Microservices architecture: Decompose the entire system into multiple independent service modules, with each service responsible for a specific function (such as data collection, rule processing, cache management, etc.). This not only facilitates maintenance and upgrade but also enables flexible expansion of system functions according to needs. Integration with big data analysis platforms: Integrate this method with existing big data analysis platforms (such as Hadoop, Spark), utilize their powerful data processing capabilities and rich analysis tools to deeply mine historical data and discover potential patterns and trends.

[0072] If there is no cache object, initialize a metric calculation object containing the metric value, start timestamp, calculation type, atomic metric encoding, and calculation rule, and store it in the cache.

[0073] Specifically, the metric value: This is a numerical value calculated according to the specific atomic metric type (such as boot time, cumulative consumption, etc.). For newly initialized objects, the initial value is usually 0 or set according to the actual situation.

[0074] Start timestamp: Records the time point when the attribute data was first received, used for calculating the duration or other calculations that depend on the time interval. For example, when calculating the device boot time, the start timestamp marks the time when the device was first recorded as being in the boot state.

[0075] Calculation Type: Specifies how to process and calculate metric values. According to the document description, it may include the following types:

[0076] Aggregation Calculation: Such as summation, average, maximum value, minimum value, etc.

[0077] Duration Calculation: Determine whether to update the duration of the current period according to specific rules.

[0078] Accumulated Value Calculation: Obtain the result by subtracting the previous value from the current value and then performing an aggregation calculation.

[0079] Custom Rule Calculation: Allows users to write simple rules through JS scripts for calculation, but note to avoid overly complex logic to reduce latency in streaming calculations.

[0080] Atomic Metric Encoding: This is a unique identifier for each atomic metric, used to distinguish different calculation tasks and rule configurations. For example, "MACHINE_ON_TIME" represents the atomic metric for calculating the device's power-on duration.

[0081] Calculation Rule: Contains the detailed rules required to perform specific calculations. For duration calculation, this may involve when to start timing, when to stop timing, etc.; for custom rule calculation, it may be the specific JS script content.

[0082] After creating the metric calculation object composed of all the above information, use "Property Unique Encoding - Atomic Metric Encoding" as the key and write this object into the cache. The cache tool mentioned here is Caffeine, which is a high-performance Java cache library suitable for reactive streaming data processing scenarios.

[0083] The purpose of doing this is to ensure that when receiving data with the same property next time, the existing calculation object can be directly obtained from the cache without re-initialization, thereby improving processing efficiency and ensuring data consistency and accuracy.

[0084] For example, assume that the power-on duration of an industrial device (DEVICE_001) is being monitored. The device regularly sends its operating status data (e.g., power on or off). The goal is to calculate the device's power-on duration based on these real-time data streams and save the results for subsequent analysis.

[0085] Initialize the Metric Calculation Object

[0086] Step 1: Generate the cache key

[0087] Property Unique Encoding: Assume it is DEVICE_001.

[0088] Atomic Index Encoding: The index encoding used to represent the device's power-on duration is set to MACHINE_ON_TIME.

[0089] Combine the two to generate a cache key: "DEVICE_001-MACHINE_ON_TIME".

[0090] Step Two: Check the cache

[0091] The system first attempts to find if there is a corresponding index calculation object in the cache using the cache key generated above. If not found, proceed to the next step.

[0092] Step Three: Initialize the index calculation object

[0093] Index Value: Since this is the first time the device's status data is received, the initial value of the power-on duration is set to 0.

[0094] Start Timestamp: Record the time of the currently received data as the start timestamp. For example, the current time is 2025-02-17T18:03:00Z.

[0095] Calculation Type: Set to "Duration Calculation" because we are concerned with the duration of the device's power-on state.

[0096] Atomic Index Encoding: Set to MACHINE_ON_TIME.

[0097] Calculation Rule: Define the logic for when to start and stop timing. For example, start timing when the device status changes from off to on, and stop timing and accumulate this period to the total power-on duration when the device changes from on to off.

[0098] Step Four: Store in the cache

[0099] Use the Caffeine tool to write the initialized index calculation object into the cache with "DEVICE_001-MACHINE_ON_TIME" as the key. This object contains all the necessary information (index value, start timestamp, calculation type, atomic index encoding, and calculation rule) so that when processing the same type of data for the same device later, the existing calculation object can be directly utilized to avoid repeated initialization.

[0100] Whenever the system receives new status data about DEVICE_001 again, it will first try to find the corresponding metric calculation object in the cache using the same cache key. If found, it will update the metric value according to the new data (for example, if it is in the powered-on state, continue to accumulate the duration; if it is in the powered-off state, stop accumulating and record the accumulated duration). This can ensure real-time and accurate tracking of the device's power-on duration, while reducing unnecessary database interactions and improving the system's efficiency and response speed.

[0101] Furthermore, it is possible to refine the management of metric values: When initializing the metric calculation object, different initial values can be set according to different business requirements. For example, for metrics of the cumulative value calculation type, in addition to being initialized to 0, a non-zero initial value can also be set based on historical data or other factors. Introduce a version control mechanism for metric values to ensure accurate and consistent updates to metric values in a concurrent environment. Optimize the start timestamp processing: Increase support for timestamp formats, allowing users to customize the timestamp format (such as the ISO 8601 standard) to adapt to the data format requirements from different sources. Implement validation of the timestamp's validity to ensure that all data entering the system has valid timestamp information and avoid calculation errors caused by invalid or incorrect timestamps. Enhance the flexibility of calculation types: Provide more detailed calculation type classifications and configuration options, enabling users to define calculation logic more precisely. For example, support aggregation calculations based on time periods, allowing users to specify specific time periods for data aggregation. For complex calculation types, provide preset templates or wizard tools to help users quickly configure calculation rules, reducing the workload and error probability of manual configuration. Dynamic management of atomic metric encoding: Allow the dynamic generation and modification of atomic metric encoding to adapt to the needs of business changes. For example, when adding new devices or adjusting monitoring metrics, new atomic metric encoding can be created and applied to the system conveniently. Implement version management and rollback functions for atomic metric encoding so that it can be quickly restored to the previous stable state when problems occur after encoding changes. Refine the description and execution of calculation rules: Support writing calculation rules in multiple languages, not limited to JavaScript scripts, but also including other programming languages such as Python, to meet the needs of users with different development backgrounds. Enhance the function of the rule engine so that it can support more complex conditional judgments and logical operations, improving the application scope and flexibility of calculation rules.

[0102] If there exists the cache object, perform the corresponding metric calculation operation according to the calculation type, where the metric calculation operation includes duration calculation, cumulative value calculation, aggregation calculation, or custom rule calculation, and update the metric value in the cache.

[0103] Specifically, for the duration calculation: When the calculation type is duration, the system calculates the duration of a specific state (such as the power-on state) based on the changes in the device status.

[0104] For example, when monitoring the power-on duration of a device, if the previous value and the current value are both in the power-on state, the difference between the current timestamp and the start timestamp is calculated and this time is accumulated into the metric value. If the previous value is in the power-off state and the current value changes to the power-on state, the start timestamp is updated to the current timestamp and the previous value is changed to power-on. Conversely, if it changes from power-on to power-off, the current state is recorded as power-off and the duration accumulation stops.

[0105] Aggregate value calculation: Aggregate value calculation involves aggregating the result of subtracting the previous value from the current value. This is typically used to track the change in a value over a period of time.

[0106] For example, to calculate the energy consumption increment of a device over a period of time, the new data received each time is subtracted from the previous data, and this difference is then accumulated into the total consumption.

[0107] Aggregation calculation: Aggregation calculations include operations such as summation, average, maximum, minimum, etc. Such calculations are often used to summarize data over a period of time to obtain overview statistical information.

[0108] For example, to obtain the average temperature in a day, all received temperature data points are summed and then divided by the number of data points; or to find the highest or lowest temperature.

[0109] Custom rule calculation: Custom rules allow users to write specific logic to process data through a scripting language (such as JavaScript). This approach provides great flexibility and allows complex calculation rules to be defined according to specific business requirements.

[0110] The rules should be kept as simple as possible to avoid overly complex logic to reduce latency in streaming calculations. For example, an alarm or notification can be triggered based on certain conditions, or a transformation formula can be applied to data in a specific pattern.

[0111] Update the metric value in the cache

[0112] After performing any of the above types of calculations, the system needs to rewrite the updated metric value back to the cache. This is to ensure that the latest calculation results can be used for iterative calculations when processing the same type of data next time.

[0113] For the duration calculation, it may also be necessary to update the start timestamp and the status of the previous value (such as power-on / power-off) to accurately reflect the changes in the device status.

[0114] Before bulk writing to the database is completed, all updates are performed in the cache, which can ensure real-time performance and reduce the pressure on the database.

[0115] For example, assume that the startup duration and cumulative energy consumption of a device (DEVICE_001) are being monitored. The device periodically sends its operating status (such as on / off) and current energy consumption data. The goal is to calculate the startup duration and cumulative energy consumption of the device based on these real-time data streams and save the results for subsequent analysis.

[0116] Duration calculation (startup duration)

[0117] Check the cache:

[0118] Assume that the cache key is "DEVICE_001-MACHINE_ON_TIME" and the system finds an existing cache object.

[0119] Get the information of the cache object:

[0120] The cache object contains the following information:

[0121] Metric value: Cumulative startup duration (e.g., 3 hours)

[0122] Start timestamp: The time when the device was last received in the on state (e.g., 2025-02-17T18:03:00Z)

[0123] Calculation type: Duration calculation

[0124] Previous value: On state

[0125] Receive new data:

[0126] The new data indicates that the device is still in the on state and the current time is 2025-02-17T21:04:00Z.

[0127] Calculate and update the cache:

[0128] Since the device status has not changed (still on), calculate the duration from the last recorded time to the present (i.e., 2025-02-17T21:04:00Z - 2025-02-17T18:03:00Z = 3 hours 1 minute).

[0129] Add this time period to the existing cumulative startup duration (3 hours + 3 hours 1 minute = 6 hours 1 minute).

[0130] Update the metric value in the cache to 6 hours 1 minute and keep the start timestamp unchanged (because the device is still on).

[0131] Accumulated value calculation (cumulative energy consumption)

[0132] Check cache:

[0133] Assume the cache key is "DEVI CE_001-TOTAL_ENERGY_CONSUMPT ION", and the system finds an existing cache object.

[0134] Get information of the cache object:

[0135] The cache object contains the following information:

[0136] Metric value: Cumulative energy consumption (e.g., 100 kWh)

[0137] Previous value: The last recorded energy consumption value (e.g., 90 kWh)

[0138] Receive new data:

[0139] The new data indicates that the current energy consumption is 110 kWh.

[0140] Calculate and update the cache:

[0141] Calculate the difference between the current energy consumption and the previous value (110 kWh - 90 kWh = 20 kWh).

[0142] Add this difference to the cumulative energy consumption (100 kWh + 20 kWh = 120 kWh).

[0143] Update the metric value in the cache to 120 kWh and update the previous value to the current energy consumption value (110 kWh).

[0144] Aggregate calculation (average energy consumption)

[0145] Check cache:

[0146] Assume the cache key is "DEVI CE_001-AVERAGE_ENERGY_CONSUMPT ION", and the system finds an existing cache object.

[0147] Get information of the cache object:

[0148] The cache object contains the following information:

[0149] Metric value: Cumulative energy consumption (e.g., 120 kWh)

[0150] Number of data points: The number of recorded energy consumption data points (e.g., 3)

[0151] Receive new data:

[0152] The new data indicates that the current energy consumption is 115 kWh.

[0153] Calculate and update the cache:

[0154] Add the new energy consumption value to the cumulative energy consumption (120 kWh + 115 kWh = 235 kWh).

[0155] The number of data points increases (3 + 1 = 4).

[0156] Calculate the new average energy consumption (235 kWh / 4 = 58.75 kWh).

[0157] Update the metric value in the cache to 58.75 kWh, and update the cumulative energy consumption and the number of data points.

[0158] Furthermore, state transition detection can also be performed: when the device state changes (e.g., from powered on to powered off), accurately record the time points of the state changes, and accurately calculate the duration of each state based on these time points. Introduce a state change history recording mechanism to save the timestamps and state values of each state change for subsequent analysis. Multi-state support: Support the simultaneous calculation of the durations of multiple states, not just limited to two states (such as powered on / powered off). For example, the time length of the device in the standby mode can also be tracked. Difference verification: Before calculating the cumulative value, add a step to verify the reasonableness of the difference between the new and old data to prevent incorrect accumulation caused by abnormal data. If abnormal data (such as negative energy consumption) is found, it can be decided whether to ignore the data or trigger an alarm notification according to the preset policy. Periodic reset: Set the reset period of the cumulative value according to business requirements (such as resetting at zero o'clock every day) to ensure that the cumulative value can reflect the change trend within a specific time period. Dynamic window support: Implement aggregation calculation methods based on sliding windows or fixed windows, allowing users to adjust the window size as needed to obtain data summary results at different granularities. Provide multiple aggregation function options (such as median, percentile, etc.) to enrich the data summary methods. Grouping aggregation: Support grouping data according to specific attributes (such as device type, geographical location, etc.) and then performing aggregation calculations to make the analysis results more targeted. Rule template library: Establish a library containing common rule templates, allowing users to directly select applicable templates to quickly configure custom rules, reducing the workload of manually writing scripts. For complex rules, provide a graphical editor to help users build a logic flow chart by dragging and dropping components, simplifying the rule creation process. Rule testing and simulation: Before formal application, provide a test environment for the rules, allowing users to input sample data to verify the correctness and expected effects of the rules. Support the rule simulation running function to preview the results of rule execution without affecting the actual production data.

[0159] Batch write the metric values in the cache to the target database according to a preset time period, and use the timestamp and the atomic metric code as the unique identifier to avoid duplicate writing.

[0160] Specifically, set the write cycle: The system supports setting different write cycles (such as every minute, every hour, or at midnight), which means that data in the cache can be batch-written to the database at regular time intervals (e.g., every minute, every hour).

[0161] Selecting an appropriate write cycle is crucial for balancing real-time performance and system load. A shorter cycle can provide more near-real-time data updates, but may increase the system's I / O burden; a longer cycle may affect data timeliness.

[0162] Batch processing: At the end of each write cycle, the system collects all cache items to be written and sends them to the database for writing in one go.

[0163] Batch writing helps reduce the number of database connections and transaction commits, thereby improving overall write efficiency and reducing the pressure on the database.

[0164] Generate a unique identifier: For each metric value to be written to the database, the system uses a timestamp plus an atomic metric code to create a unique identifier. For example, "2025-02-17T18:15:00Z-MACHINE_ON_TIME" can be used as the unique identifier for the power-on duration of a certain device at a specific moment.

[0165] This combination ensures that even if multiple different types of metric values need to be written to the database within the same time period, they will not conflict or overwrite each other.

[0166] Avoid duplicate writing: Before writing, the system can prevent duplicate writing by checking whether the unique identifier already exists in the database. If data with the same identifier is found to exist, the system can choose to skip this write or perform an update operation according to the business logic.

[0167] This mechanism effectively avoids data redundancy problems caused by overlapping scheduled tasks or other reasons, ensuring data consistency and accuracy in the database.

[0168] For example, assume that we want to monitor the power-on duration of an industrial device (DEVICE_001). This device regularly sends its operating status data (e.g., power on or off). The goal is to calculate the power-on duration of the device based on these real-time data streams and save the result to the database at the end of each hour.

[0169] Step 1: Set the write cycle and prepare the data

[0170] Write cycle: Set to write once at the whole hour every hour.

[0171] Current time: 2025-02-17T18:00:00Z, which means the system will perform the next batch write operation at 2025-02-17T19:00:00Z.

[0172] Step Two: Data Processing in the Cache

[0173] During the period from 2025-02-17T18:00:00Z to 2025-02-17T19:00:00Z, the system continuously updates the metric values in the cache according to the received data. For example, during this period, the cumulative device on-time is 3 hours.

[0174] Step Three: Generate a Unique Identifier and Prepare for Batch Write

[0175] When it reaches 2025-02-17T19:00:00Z, the system prepares to batch write the data in the cache to the database.

[0176] For the metric of on-time, a unique identifier is generated using the timestamp plus the atomic metric code. In this example, the unique identifier is: "2025-02-17T18:00:00Z-MACHINE_ON_TIME".

[0177] Step Four: Check if the Unique Identifier Exists and Perform the Write

[0178] Query the database: The system first queries the database to check if there is a record with the same unique identifier (i.e., "2025-02-17T18:00:00Z-MACHINE_ON_TIME").

[0179] If no matching record is found, proceed to the next step; if a matching record is found, decide whether to overwrite the old data or skip this write according to the business requirements.

[0180] Perform the write operation: Assuming no record with the same unique identifier is found in the database, the system writes the following information to the database:

[0181] Timestamp: 2025-02-17T18:00:00Z

[0182] Atomic metric code: MACHINE_ON_TIME

[0183] Metric value: 3 hours (representing the cumulative on-time during this period)

[0184] Other possible information: such as device ID (DEVICE_001), source, etc.

[0185] Reset the cache: After the write is complete, the system resets the corresponding metric value in the cache to zero, ready for data collection and calculation in the next cycle.

[0186] Furthermore, the write cycle can be dynamically adjusted: According to changes in system load or business requirements, the write cycle is dynamically adjusted. For example, shorten the write cycle during low system load to improve real-time performance; extend the write cycle during peak periods to relieve database pressure. Implement an event-triggered write mechanism. In addition to scheduled writes, when certain key events occur (such as device status changes), the write operation is immediately triggered. Support multiple time granularities: Provide more detailed time granularity options, allowing users to select different write cycles according to their needs (such as minute-level, hour-level, day-level, etc.) to adapt to data storage requirements in different scenarios. Multi-dimensional unique identifiers: Based on using timestamps and atomic metric encodings as unique identifiers, more dimensions of information are added to enhance uniqueness. For example, device IDs, geographical location information, etc. can be added to form a more complex composite key (such as "2025-02-17T18:00:00Z-MACHINE_ON_TIME-DEVICE_001"), thereby further reducing the risk of conflicts. Automatic detection and resolution of conflicts: When the situation of the same unique identifier is found, it is automatically detected and a solution is provided. For example, data can be merged, old data can be overwritten, or a conflict log can be recorded for subsequent manual review. Intelligent batch writing: For large amounts of data, an intelligent batch strategy is adopted, and the size of each write data packet is dynamically adjusted according to the amount of data, ensuring that the network bandwidth can be fully utilized without overloading the database. Transaction management and rollback mechanism: Introduce a transaction management mechanism to ensure that the entire batch write operation is either all successful or all failed, maintaining data consistency and integrity. Implement an efficient rollback mechanism that can quickly restore to the state before the write in case of an exception, reducing the impact on the system. Data validity verification: Before writing, the data in the cache is verified for validity, excluding invalid or incorrect data to ensure that only high-quality data is written to the database. Data cleaning rules: Define a series of data cleaning rules to automatically correct data with incorrect formats or minor errors, improving data quality and usability.

[0187] Reset the metric value in the cache to prepare for the next calculation cycle.

[0188] Specifically, clear the old values: After successfully writing the metric value in the cache to the target database, the system needs to clear or reset these processed data values. For example, the cumulative power-on duration calculated within an hour (assumed to be 3 hours), after writing it to the database, this cumulative value should be set to 0 to start a new timing.

[0189] Retain necessary information: Although the main metric values will be reset, certain status information may need to be retained to support continuous calculations. For example, when calculating the duration, if the device is still powered on at the end of the cycle, the current start timestamp and the previous value (such as the powered-on state) need to be saved. This enables seamless continuation of the previous calculation logic at the start of the new cycle.

[0190] Update timestamps: If calculations based on time periods are involved, relevant timestamps also need to be updated. For example, for a calculation task that runs once an hour, after completing the task for the current cycle and resetting the cache, the start timestamp of the new cycle should be recorded (e.g., changing from 2025-02-17T18:00:00Z to 2025-02-17T19:00:00Z).

[0191] Initialize calculation objects for the new cycle: For certain types of calculations (such as aggregation calculations, custom rule calculations, etc.), specific calculation objects or parameters may need to be re-initialized. For example, at the start of a new calculation cycle, certain configuration items may need to be adjusted according to the latest business requirements or rule scripts may need to be reloaded.

[0192] For example, the goal is to monitor the power-on duration of DEVICE_001, write the accumulated power-on duration data to the database at the end of each hour, and then reset the cache to prepare for the next calculation cycle.

[0193] Step 1: Initialization and data collection

[0194] Current time: 2025-02-17T18:00:00Z

[0195] Cache object information:

[0196] Metric value: 0 (initial power-on duration)

[0197] Start timestamp: 2025-02-17T18:00:00Z (time of first record)

[0198] Previous value: powered-off state

[0199] Calculation type: duration calculation

[0200] Atomic metric code: MACHINE_ON_TIME

[0201] Step 2: Real-time data processing

[0202] Over the next hour, the device sent multiple status updates:

[0203] 2025-02-17T18:15:00Z: The device changed from powered-off to powered-on.

[0204] Update cache: Set the start timestamp to 2025-02-17T18:15:00Z, and the previous value to the powered-on state.

[0205] 2025-02-17T18:45:00Z: The device is still in the powered-on state.

[0206] The cumulative powered-on duration increases by 30 minutes (from 2025-02-17T18:15:00Z to 2025-02-17T18:45:00Z), and the metric value is updated to 30 minutes.

[0207] 2025-02-17T19:00:00Z: The device is still in the powered-on state.

[0208] The cumulative powered-on duration increases by another 15 minutes (from 2025-02-17T18:45:00Z to 2025-02-17T19:00:00Z), and the final metric value is 45 minutes.

[0209] Step 3: Batch write to the database

[0210] Current time: 2025-02-17T19:00:00Z

[0211] Unique identifier: "2025-02-17T18:00:00Z-MACHINE_ON_TIME"

[0212] Metric value: 45 minutes

[0213] Execute the write operation: Write the above information to the database.

[0214] Step 4: Reset the cache

[0215] Clear the old value:

[0216] Reset the metric value in the cache to 0.

[0217] Retain necessary information:

[0218] Since the device is still in the powered-on state at 2025-02-17T19:00:00Z, the current status information needs to be retained.

[0219] Start timestamp: Update to 2025-02-17T19:00:00Z

[0220] Previous value: Remain the powered-on state

[0221] Update timestamp:

[0222] Set the start timestamp of the new cycle to 2025-02-17T19:00:00Z to record data for the next hour.

[0223] Cached object information after reset

[0224] Metric value: 0 (new calculation cycle starts, re-timing)

[0225] Start timestamp: 2025-02-17T19:00:00Z (start time of new cycle)

[0226] Previous value: power-on state (device state unchanged)

[0227] Calculation type: duration calculation

[0228] Atomic metric code: MACHINE_ON_TIME

[0229] In some embodiments of the present application, before generating a cache key based on the attribute unique code and the atomic metric code and querying whether there is a corresponding metric calculation object in the cache, it includes:

[0230] Real-time collect device data and perform formatting processing, extract data reporting time and attribute unique code;

[0231] Subscribe to data streams according to atomic metric configuration rules and receive real-time data.

[0232] Specifically, real-time data collection: Purpose: To obtain the latest status or performance data from various devices in real time. Implementation method: The system needs to be equipped with a real-time data collection module, which is responsible for communicating with external devices and collecting status reports or other forms of data streams sent by them. For example, in an industrial Internet of Things (IoT) environment, this may involve connecting to a sensor network, a PLC (programmable logic controller), a SCADA system, etc.

[0233] Formatting processing: Purpose: To ensure that all received data follows a consistent structure and standard for subsequent processing. Implementation method: Once the data is collected, it needs to be standardized and formatted. This usually includes converting the original data into an easily parsable format (such as JSON, XML), and extracting key information fields, such as data reporting time, attribute unique code, etc. In addition, it may also include data cleaning steps, such as removing outliers, filling in missing values, etc., to improve data quality.

[0234] Extract data reporting time and attribute unique code

[0235] Data reporting time: This refers to the specific timestamp when each data point is recorded or sent. It is key information for tracking the order of events and is particularly important for time series analysis.

[0236] Attribute Unique Encoding: Each piece of data should be associated with a specific physical entity or logical concept. The attribute unique encoding can clearly identify the data source. For example, in the case of monitoring multiple devices, each device has its own ID, and different sensors on each device may also have their own encodings.

[0237] Subscribe to the data stream according to the atomic metric configuration rules and receive real-time data

[0238] Atomic Metric Configuration Rules: Definition: An atomic metric is a basic unit of measurement that describes the state of a system, such as the boot duration, average temperature, etc. The atomic metric configuration rules define which types of data should be concerned and how to process this data. Configuration content: including but not limited to selecting specific attributes to be monitored (such as temperature, pressure), determining the calculation method (such as summation, average), setting trigger conditions (such as exceeding a certain threshold), etc.

[0239] Subscribe to the data stream: Purpose: According to the predefined atomic metric configuration rules, subscribe to relevant real-time data streams to ensure that only the information useful for the current analysis task is received. Implementation method: Usually implemented through a message queue or a publish / subscribe mode. For example, using message middleware such as Kafka, RabbitMQ, allowing the system to subscribe to the topics of interest, and automatically receiving new data when it arrives.

[0240] Receive real-time data: Process: Once the subscription is set up, the system can start receiving real-time data streams from various sources. These data will then be passed to downstream components for further processing, such as generating cache keys, querying caches, performing calculations, etc.

[0241] In some embodiments of the present application, the calculation type includes at least one of the following:

[0242] Aggregation calculation, including summation, average value, maximum value or minimum value;

[0243] Duration calculation, accumulating the total duration according to the difference in state change timestamps;

[0244] Accumulated value calculation, accumulating by the difference between the current value and the previous value;

[0245] Custom rule calculation, performing user-defined numerical operations based on a script engine.

[0246] Specifically, aggregation calculation

[0247] Definition: Aggregation calculation refers to the operation of summarizing and statistically analyzing a set of numerical values, usually used to obtain the overall characteristics of a set of data.

[0248] Sum: Adding together all relevant numerical values to obtain a total. For example, calculating the total energy consumption of a device over a period of time.

[0249] Average (AVG): Calculating the arithmetic mean of a set of numerical values. For example, when monitoring the temperature readings of multiple sensors, the average temperature over a period of time can be calculated to understand the environmental change trend.

[0250] Maximum (MAX): Identifying the largest value in a set of numerical values. This is very useful for identifying peak performance or abnormal high points, such as determining the highest electricity consumption in a day.

[0251] Minimum (MIN): Identifying the smallest value in a set of numerical values. Similar to the maximum, but focuses on the lowest point, such as the lowest temperature record.

[0252] Duration Calculation

[0253] Definition: Duration calculation is to calculate the total time length that a certain state persists based on the timestamps of state changes.

[0254] Specific Implementation:

[0255] The system needs to track the timestamps of each state change and update the relevant time information when the state changes. For example, if we want to calculate the power-on duration of a device, whenever the device state changes from off to on, the system will record the start timestamp; when the device state changes back to off again, the system will calculate the time difference and accumulate it into the total power-on duration.

[0256] This calculation is particularly applicable to application scenarios that require monitoring the occurrence frequency or duration of certain events, such as device running time, service availability, etc.

[0257] Accumulated Value Calculation

[0258] Definition: Accumulated value calculation is a process of cumulative calculation by comparing the difference between the current value and the previous value.

[0259] Specific Implementation:

[0260] First, the system needs to store the previously received data value as the "previous value".

[0261] When new data arrives, the system calculates the difference between the current value and the previous value and adds this difference to the accumulated total.

[0262] This method is often used to track quantities that gradually increase or decrease over time, such as the increase in energy consumption, changes in inventory levels, etc.

[0263] Custom Rule Calculation

[0264] Definition: Custom rule calculation allows users to define complex calculation logics by writing scripts to meet specific business requirements.

[0265] Specific implementation:

[0266] Users can write scripts (such as JavaScript) according to actual needs and execute these scripts through the built-in script engine.

[0267] These scripts can include conditional judgments, loop structures, and other programming elements to perform more complex numerical operations or decision-making processes.

[0268] For example, dynamically adjust the alarm threshold based on the working mode of the device, or execute complex mathematical model predictions of future trends based on multi-dimensional data inputs.

[0269] Although it has extremely high flexibility, attention should be paid to keeping the code concise when designing custom rules to avoid increased latency caused by overly complex logics.

[0270] In some embodiments of the present application, the specific steps of the duration calculation include:

[0271] When it is detected that the device state changes from shutdown to startup, record the current timestamp as the start time;

[0272] When the device state remains startup, accumulate the difference between the current timestamp and the start time to the metric value;

[0273] When the state changes to shutdown, stop accumulating and record the final metric value.

[0274] Specifically, when it is detected that the device state changes from shutdown to startup, record the current timestamp as the start time

[0275] Purpose: Determine the time point when the device enters the startup state as the starting point for calculating the startup duration.

[0276] Operation details: The system receives device status updates in real time. When the system detects that the device status changes from shutdown to startup, immediately obtain the current timestamp (such as 2025-02-17T18:29:00Z) and record this timestamp as the "start time".

[0277] At the same time, the previous value (i.e., the previous state of the device) can be updated to "startup".

[0278] Example: Assume that at 2025-02-17T18:29:00Z, the system receives a status update of DEVICE_001 indicating that the device changes from shutdown to startup, then the system records this time as the start time.

[0279] When the device status remains powered on, accumulate the difference between the current timestamp and the start time to the metric value

[0280] Purpose: While the device remains powered on, continuously accumulate its running time.

[0281] Operation details: Each time new status data is received, if it is found that the device is still powered on, calculate the difference between the current timestamp and the previously recorded start time. Accumulate this difference into the existing metric value (initially 0). For example, if the current time is 2025-02-17T18:35:00Z, the accumulated time is 6 minutes (from 2025-02-17T18:29:00Z to 2025-02-17T18:35:00Z). Update the metric value in the cache to the latest accumulated total duration.

[0282] Example: Assume that at 2025-02-17T18:35:00Z, the system receives the device status again and confirms that the device is still powered on. Then calculate the power-on duration during this period as 6 minutes and accumulate it into the metric value (assuming this is the first accumulation, so the metric value is 6 minutes at this time).

[0283] When the status changes to powered off, stop accumulating and record the final metric value

[0284] Purpose: When the device status changes to powered off, stop accumulating the power-on duration and save the final accumulated result for subsequent use or analysis.

[0285] Operation details: When the system detects that the device status changes from powered on to powered off, stop accumulating the power-on duration. The final accumulated duration is the current metric value, representing the total duration that the device has been powered on during this period. You can choose to immediately write this final metric value to the database or wait for the next batch write cycle (according to the preset time period). At the same time, update the previous value in the cache to "powered off" and prepare to handle the next possible status change.

[0286] Example: Assume that at 2025-02-17T19:00:00Z, the system receives the device status update indicating that the device has changed from powered on to powered off. Then the system stops accumulating the power-on duration. Assume that the accumulated power-on duration during this period is 31 minutes (from 2025-02-17T18:29:00Z to 2025-02-17T19:00:00Z). Then record these 31 minutes as the final metric value and prepare to reset the cache for the next calculation cycle.

[0287] In some embodiments of the present application, the preset time period is at least one of whole minutes, whole hours, or zero o'clock.

[0288] Specifically, minute - end: It means that at the end of each minute (i.e., every 60 seconds), the system performs a predetermined operation. For example, at times such as 2025 - 02 - 17T18:01:00Z, 2025 - 02 - 17T18:02:00Z, etc. Application scenarios: It is applicable to scenarios that require high real - time performance, such as high - frequency trading monitoring, real - time performance analysis, etc. Since a write operation is triggered every minute, it can ensure a relatively high data update frequency, but at the same time, it will increase the I / O burden on the system.

[0289] Hour - end: It means that at the end of each hour (i.e., every 3600 seconds), the system performs a predetermined operation. For example, at times such as 2025 - 02 - 17T19:00:00Z, 2025 - 02 - 17T20:00:00Z, etc. Application scenarios: It is suitable for most conventional monitoring and data analysis requirements. Compared with the minute - end mode, the hour - end mode reduces the frequency of write operations, thus reducing the pressure on the database, while still providing relatively high real - time performance.

[0290] Mid - night: It means that at the end of each day (i.e., at midnight), the system performs a predetermined operation. For example, at the time of 2025 - 02 - 18T00:00:00Z. Application scenarios: It is mainly used for scenarios that do not require real - time updates, such as end - of - day summaries, report generation, etc. This method minimizes the number of database writes and is very suitable for processing tasks of aggregating a large amount of historical data.

[0291] Real - time requirement: If the business scenario has a high requirement for data real - time performance, a shorter cycle can be selected, such as minute - end or hour - end; conversely, if real - time performance is not the primary consideration, a longer cycle can be selected, such as midnight.

[0292] System load management: Frequent data writes will increase the I / O burden on the system and may lead to performance degradation. Therefore, when selecting the preset time period, the overall load capacity of the system also needs to be considered. For high - load systems, appropriately extending the write cycle can effectively relieve the pressure.

[0293] Data volume size: The data volumes generated by different business scenarios vary greatly. For small data volumes, even using a shorter write cycle will not have too much impact on the system; while for large data volumes, a longer write cycle may need to be selected to avoid data accumulation.

[0294] In some embodiments of the present application, when initializing the metric calculation object, if it is an accumulative value calculation, the initial metric value is set to 0, and the previous value is filled according to the received data.

[0295] Specifically, initialize the metric calculation object

[0296] When the system first receives data for a certain attribute and needs to calculate its cumulative value, it will first initialize an index calculation object. This object contains multiple fields for storing various information related to this attribute, so as to perform calculations and updates later.

[0297] The initial index value is set to 0

[0298] Purpose: Set a starting point so that subsequent cumulative operations have a clear basis.

[0299] Implementation method: When initializing the index calculation object, set the "index value" field to 0. This means that from the current moment on, any new data will be accumulated based on this initial value.

[0300] This choice of the initial value is reasonable because it represents a neutral point and will not affect subsequent calculation results. For example, if we want to calculate the energy consumption increment of a certain device, starting from 0 means that we only focus on the total consumption since the system started or was reset.

[0301] Fill in the previous value according to the received data

[0302] Purpose: Record the latest known state as a reference point for the next calculation.

[0303] Implementation method: When the system receives new data, in addition to updating the index value, it is also necessary to save the value of the current data point as the "previous value". This is to accurately calculate the difference between the two when new data is received next time.

[0304] For example, in the case of monitoring the energy consumption of a device, assume that the data received for the first time indicates an energy consumption of 50 kWh. Then we save 50 kWh as the previous value. When new energy consumption data (such as 60 kWh) is received next time, the energy consumption increment during this period can be calculated by comparing these two values (60 kWh - 50 kWh = 10 kWh).

[0305] In some embodiments of the present application, the target database includes at least one of a relational database, a non-relational database, or a time-series database.

[0306] Specifically, a relational database (RDBMS) is a data storage method based on tables, which uses rows and columns to organize data and describes the table structure and the relationships between fields through a predefined schema.

[0307] Features: Structured data storage: Ideal for storing structured data, with a clear structure definition for each table. Transaction support: Provides ACID (Atomicity, Consistency, Isolation, Durability) properties to ensure data consistency and reliability. Query language: Usually uses SQL (Structured Query Language) as the query language, facilitating complex query operations.

[0308] Applicable scenarios: Very suitable when complex join queries, transaction management, and strict data consistency requirements are needed.

[0309] For example, it is used to store data with a clear structure such as user information and order details.

[0310] Non-relational databases (NoSQL) are designed to store unstructured or semi-structured data, do not enforce a fixed schema, and provide higher flexibility than traditional relational databases.

[0311] Features: High scalability: Can easily scale horizontally to handle large amounts of data and high concurrent requests. Flexibility: Does not require a predefined schema and can dynamically adjust the data model to adapt to rapidly changing requirements. Diverse data models: Include document-based (such as MongoDB), key-value pairs (such as Redis), column families (such as Cassandra), graph databases (such as Neo4j), etc.

[0312] Applicable scenarios: Particularly useful for scenarios where the data structure is complex and variable and high-performance reading and writing are required, such as social networks and content management systems. Can handle large-scale data sets without sacrificing performance.

[0313] Time series databases are a type of database optimized specifically for time series data, dedicated to storing collections of data points with timestamps.

[0314] Features: Efficient time series data processing capabilities: Optimized for data arranged in chronological order, supporting efficient insert, query, and aggregation operations. Compression algorithms: To save storage space, special compression techniques are usually adopted to reduce the storage requirements for duplicate data. Built-in analysis functions: Provide powerful data analysis tools to facilitate users' trend analysis, prediction, etc.

[0315] Applicable scenarios: Particularly suitable for applications such as Internet of Things (IoT) device monitoring, financial transaction records, and environmental monitoring that require long-term storage and analysis of time series data. Can effectively manage large amounts of data that change over time and support real-time queries and historical data analysis.

[0316] In some embodiments of the present application, subscribing to the data stream according to the atomic metric configuration rule and receiving real-time data specifically includes:

[0317] Dynamically generate data subscription rules according to the described atomic metrics configuration, and receive data through a message queue or real-time data stream.

[0318] Specifically, dynamically generate data subscription rules

[0319] Atomic metrics refer to those basic measurement units that can directly reflect the system state, such as the device's power-on duration, cumulative energy consumption, etc.

[0320] Configuration content: including but not limited to selecting specific attributes to be monitored (such as temperature, pressure), determining calculation methods (such as summation, averaging), setting trigger conditions (such as exceeding a certain threshold), etc.

[0321] Dynamically generate subscription rules

[0322] Purpose: To ensure that the system can automatically adjust its data collection strategy according to the latest atomic metrics configuration to adapt to changing business requirements.

[0323] Implementation method: The system first parses the atomic metrics configuration, extracts the data attributes and corresponding filtering conditions that need to be concerned about.

[0324] Based on this information, the system automatically generates corresponding data subscription rules. For example, if the atomic metrics configuration specifies that the power-on state of the device needs to be monitored, the subscription rule may include the device ID and the status field.

[0325] These rules can be encoded into a specific message format or API call parameters for interaction with external data sources.

[0326] Receive data through a message queue or real-time data stream

[0327] A message queue is a method of asynchronous communication that allows applications to exchange data by sending and receiving messages.

[0328] Application scenario: In the Internet of Things (IoT) environment, a large amount of data generated by devices can be transmitted to the background processing system through a message queue.

[0329] The advantage of using a message queue is that it can smooth traffic peaks and ensure the stability and reliability of the system.

[0330] Implementation details: The system subscribes to a specific topic or queue and waits for messages from devices or other data sources to arrive.

[0331] When a new message arrives, the system processes it according to the predefined logic, such as parsing the message content, updating the metric values in the cache, etc.

[0332] Real-time data stream refers to the process of continuously generating and transmitting data, which is usually used in application scenarios that require immediate response.

[0333] Application scenarios: For situations that require quick response, such as financial transaction monitoring, industrial automation control, etc., using real-time data stream technology can ensure that data can be processed within the shortest time.

[0334] Implementation details: The system connects to a data stream processing platform (such as Apache Kafka, Apache Flink), and listens to the specified data stream in real time. Each record in the data stream represents a data snapshot at a certain point in time, and the system can process it immediately without waiting for the entire batch to complete.

[0335] A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the steps in the atomic metric generation method based on real-time data stream processing.

[0336] An electronic device, including a memory, a processor, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps in the atomic metric generation method based on real-time data stream processing.

[0337] Embodiment 2

[0338] The method proposed in this application requires the platform to be equipped with a real-time data acquisition module and a rule engine module. The data acquisition module is responsible for real-time acquisition of device data, performs preliminary formatting and standardization processing, and then transmits it to the rule engine module, requiring the data to carry the data reporting time and the unique encoding of the attribute to which the data belongs. The rule engine module filters and processes the data according to the atomic metric requirements, extracts the data required for atomic metric calculation, and performs subsequent atomic metric calculation.

[0339] The atomic metric calculation module is divided into four parts: a data subscription module, a metric value caching part, a calculation logic part, and a part that supports writing at the whole hour. The specific calculation steps are as follows, combined with Figure 2 .

[0340] 1. Data subscription

[0341] This module first designs SQL according to the attribute data required for atomic metric calculation, then registers it in the rule engine, and subscribes to relevant topics to obtain data, which usually involves connecting to a real-time data stream or message queue.

[0342] 2. Calculation logic

[0343] After the system receives the data, it first checks whether there is an atomic metric calculation cache based on this attribute data in the cache.

[0344] a) Without cache

[0345] If there is no cache, the system will initialize a new metric calculation object. This object contains the following main fields:

[0346] Metric value

[0347] Start timestamp

[0348] Calculation type (aggregation, duration, custom rule)

[0349] Atomic metric code

[0350] Duration calculation rule or custom rule

[0351] Previous value

[0352] The metric value, start timestamp, and previous value are filled according to the received data. When accumulating values, the metric value is 0; the calculation type (with a suffix to distinguish whether it is an accumulated value), code, and duration calculation rule or custom rule are obtained from the configuration item of the atomic metric. Then, using "attribute unique code - atomic metric code" as the key, the initialized metric calculation object is written into the metric value cache.

[0353] b) With cache

[0354] If there is a cache, the system will calculate the cumulative metric value according to the calculation type of the cache object. The calculation method depends on the specific calculation type:

[0355] Aggregation calculation, sum, average, maximum, and minimum

[0356] Duration calculation, determine whether to update the duration of the current period according to the duration calculation rule.

[0357] Accumulated value calculation, subtract the previous value from the current value and then perform aggregation calculation.

[0358] Custom rule calculation, obtain the required value according to the rule for simple rule calculation. The rule is written in JavaScript and is implemented through a JavaScript rule engine. However, the rule should not be too complex because the latency of streaming calculation needs to be considered.

[0359] After calculating the metric value, store it in the metric value cache.

[0360] 3. Cache management

[0361] The cache key is "attribute unique code - atomic metric code", and the cache object class fields have been described above. The method provided by the present invention uses the Caffeine tool for efficient cache management to adapt to reactive streaming data processing.

[0362] 4. Regular hourly batch writing

[0363] The system supports timed write operations at whole minutes, whole hours, and zero o'clock, and writes the cached atomic metric values to the target database in batches. According to the configuration of this atomic metric, the target database can write to relational databases, non-relational databases, and time series databases. This step also needs to perform the following two operations:

[0364] a) The write operation needs to ensure that the timestamp + atomic metric code is used as the unique identifier to avoid data duplication or confusion.

[0365] b) After writing, the system sets the cached metric value to zero to prepare for the next calculation cycle.

[0366] Through the above steps, the system can efficiently process real-time data, perform metric calculations, and ensure the accuracy and timeliness of the data.

[0367] Example

[0368] Such as Figure 3 is an example of calculating the startup duration of a certain device using the real-time atomic metric calculation method provided by this patent. The startup status attribute value of the device is used to determine whether the device is started (calculation rule).

[0369] When the startup status data is received for the first time, a metric calculation object is created, including:

[0370] Startup duration metric value (initial value is 0)

[0371] Start timestamp (current timestamp)

[0372] Calculation type (duration)

[0373] Atomic metric code (for example: MACHINE_ON_TIME)

[0374] Previous value (startup or shutdown)

[0375] Whenever new startup status data is received again, update the metric value according to the current status and the previous value:

[0376] If both the previous value and the current value are in the startup state, calculate the duration (current timestamp - start timestamp) and accumulate this time to the metric value.

[0377] If the previous value is in the shutdown state and the current value is in the startup state, update the start timestamp to the current timestamp and change the previous value to startup.

[0378] If the previous value is in the startup state and the current value is in the shutdown state, record the current state as shutdown and stop accumulating the duration.

[0379] If both the previous value and the current value are in the shutdown state, no operation is performed.

[0380] According to the timing time type (per minute, per hour, per day) configured by this atomic index, regularly write the calculated total startup duration into the database, and at the same time reset the index value in the cache to zero, update the start timestamp to the timestamp when starting to write in batches regularly, and start the next cycle of calculation.

[0381] What is not described in this application can be realized by adopting or referring to the existing technology.

[0382] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key point of each embodiment is to illustrate the differences from other embodiments.

[0383] The above are only the embodiments of this application and are not used to limit this application. For those skilled in the art, this application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims

1. A method for generating atomic indicators based on real-time data streaming processing, characterized in that: include: Generate a cache key based on the unique attribute code and the atomic indicator code, and query whether the corresponding indicator calculation object exists in the cache; If the cache object does not exist, initialize the indicator calculation object including the indicator value, start timestamp, calculation type, atomic indicator code and calculation rule, and store it in the cache; If the cache object exists, perform a corresponding indicator calculation operation according to the calculation type, the indicator calculation operation including duration calculation, cumulative value calculation, aggregation calculation or custom rule calculation, and update the indicator value in the cache; Writing the cached index values ​​into the target database in batches according to a preset time period, and using the timestamp and the atomic index code as a unique identifier to avoid repeated writing; The indicator value in the cache is reset to prepare for the next calculation cycle.

2. The method according to claim 1, characterized in that Before generating a cache key based on the attribute unique code and the atomic indicator code and querying whether a corresponding indicator calculation object exists in the cache, the following steps are included: Collect device data in real time and format it, extract data reporting time and unique attribute code; Subscribe to data streams according to atomic indicator configuration rules and receive real-time data.

3. The method according to claim 1, characterized in that The calculation type includes at least one of the following: Aggregate calculations, including sum, average, maximum, or minimum; Duration calculation: the total duration is generated by adding up the difference in state change timestamps; Accumulated value calculation, which accumulates the difference between the current value and the previous value; Custom rule calculation, based on the script engine to perform user-defined numerical operations.

4. The method according to claim 3, characterized in that The specific steps of calculating the duration include: When it is detected that the device status changes from shutdown to startup, the current timestamp is recorded as the start time; When the device status continues to be powered on, the difference between the current timestamp and the start time is accumulated to the indicator value; When the status changes to shutdown, stop accumulating and record the final indicator value.

5. The method according to claim 1, characterized in that The preset time period is at least one of a whole minute, an hour, or zero o'clock.

6. The method according to claim 1, characterized in that When initializing the indicator calculation object, if it is a cumulative value calculation, the initial indicator value is set to 0, and the previous value is filled according to the received data.

7. The method according to claim 1, characterized in that The target database includes at least one of a relational database, a non-relational database or a time series database.

8. The method according to claim 2, characterized in that: The data stream is subscribed according to the atomic indicator configuration rule to receive real-time data, specifically: Data subscription rules are dynamically generated according to the atomic indicator configuration, and data is received through a message queue or a real-time data stream.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the method for generating atomic indicators based on real-time data streaming processing as described in any one of claims 1 to 8 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the method for generating atomic indicators based on real-time data streaming processing as described in any one of claims 1 to 8 are implemented.

Images