Flow batch processing mode switching method and device for measurement data, equipment, storage medium and program product

By acquiring key indicator information from measurement data, using the forward linking algorithm for rule matching and switching direction determination, constructing a difference set, and executing progressive task migration, the problems of low resource utilization and poor adaptability in switching between stream and batch processing modes are solved, achieving efficient and flexible data processing.

CN121919014AActive Publication Date: 2026-04-24GUANGZHOU POWER SUPPLY BUREAU GUANGDONG POWER GRID CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
GUANGZHOU POWER SUPPLY BUREAU GUANGDONG POWER GRID CO LTD
Filing Date
2026-03-26
Publication Date
2026-04-24

AI Technical Summary

Technical Problem

Existing stream processing systems reach resource utilization bottlenecks during peak loads, leading to increased data processing latency. Batch processing systems are idle during off-peak periods, and existing mode switching systems incur high resource overhead and are time-consuming during the switching process, failing to effectively adapt to the temporal continuity characteristics of metering data.

Method used

By acquiring key indicator information from metering data, using a forward linking algorithm for rule matching, counting the number of triggers of high and low load type rules, determining the switching direction, constructing three sets of differences, executing progressive task migration, and optimizing switching parameters using a power load elastic response switching curve model, intelligent dynamic switching of the batch processing mode is achieved.

Benefits of technology

It improves the resource utilization efficiency and adaptability of the metering data processing system, ensures the consistency and accuracy of data processing, enhances the system's flexibility and configurability, and adapts to different business scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121919014A_ABST
    Figure CN121919014A_ABST
Patent Text Reader

Abstract

The invention relates to a flow batch processing mode switching method and device for metering data, computer equipment, a readable storage medium and a program product, and relates to the technical field of data processing. The processing efficiency of the system on the metering data can be improved. The method comprises the following steps: performing rule matching according to key index information and a rule condition set, counting triggering times of a high-load type rule and a low-load type rule according to a rule matching result, and determining a switching direction of a stream processing mode and a batch processing mode according to the triggering times; creating a state snapshot of the current processing mode, obtaining a state change set and a rule trigger state change set according to the state snapshot, and constructing three types of difference sets including newly added metering data, the state change set and the rule trigger state change set; progressive task migration is executed based on the three types of difference sets, and when the switching direction is switching from the stream processing mode to the batch processing mode, the progressive task migration is executed based on the optimal switching parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for switching between streaming and batch processing modes for metering data. Background Technology

[0002] With the rapid development of smart grids, a large amount of metering data is collected in real time and needs to be processed promptly, which places higher demands on data processing systems. In the field of power metering data processing, stream processing and batch processing are two main data processing modes. Stream processing can process continuously arriving data streams in real time, providing low-latency calculation results; while batch processing accumulates data into batches for centralized processing, resulting in higher throughput.

[0003] In traditional technologies, stream processing systems like Apache Flink often reach resource utilization bottlenecks when handling peak loads of electricity metering data, leading to increased data processing latency and even backpressure issues, affecting the timeliness of real-time monitoring and alarms. While batch processing systems like Apache Spark can efficiently process large volumes of data, pre-allocated computing resources remain largely idle during off-peak electricity metering data periods, resulting in resource waste. Current industry solutions, such as the Lambda architecture which deploys both stream and batch processing systems, offer the advantages of both modes, but lack an effective coordination mechanism between the two systems. Existing systems supporting mode switching, such as Spark Structured Streaming, often employ a full state migration approach during switching, failing to consider the unique temporal continuity of metering data. This results in high resource overhead and long processing times during the switching process, leading to low processing efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for switching between streaming and batch processing modes for metering data, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for switching between streaming and batch processing modes for metering data, including:

[0006] The system acquires key indicator information of newly added measurement data, and performs rule matching with high-load type rules or low-load type rules through a forward linking algorithm based on the key indicator information and a preset set of rule conditions to obtain the rule matching result. Each condition in the preset set of rule conditions includes indicator name, comparison operator, threshold, duration and trigger type attribute, wherein the trigger type attribute includes high-load type and low-load type.

[0007] Based on the rule matching results, the number of triggers corresponding to the high-load type rule and the low-load type rule are counted within a preset sliding time window, and the switching direction between stream processing mode and batch processing mode is determined based on the number of triggers.

[0008] Create a state snapshot of the current processing mode. Based on the state snapshot, obtain a state change set and a rule-triggered state change set, and construct three types of difference sets containing the newly added metering data, the state change set, and the rule-triggered state change set.

[0009] A progressive task migration is performed based on the three sets of differences, wherein when the switching direction is from the streaming mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0010] In one embodiment, determining the switching direction between stream processing mode and batch processing mode based on the number of triggers includes:

[0011] If the number of triggers of the high-load type rule within the preset sliding time window reaches the high-load trigger threshold, the stream processing mode is switched to the batch processing mode; if the number of triggers of the low-load type rule within the preset sliding time window reaches the low-load trigger threshold, the batch processing mode is switched to the stream processing mode.

[0012] In one embodiment, the state snapshot includes the intermediate result of the current operator, window aggregation data, physical timestamp, and logical sequence number, wherein the intermediate result refers to the calculation state where the measurement data has been partially processed but not yet completed, and the window aggregation data refers to the cumulative result of the number of triggers of the successfully matched high-load type rule and the low-load type rule within the preset sliding time window.

[0013] The step of obtaining the state change set and the rule-triggered state change set based on the state snapshot includes: obtaining the changes in the key indicator information based on the intermediate results, summarizing the changes to obtain the state change set; and summarizing the window aggregated data to obtain the rule-triggered state change set.

[0014] In one embodiment, the construction includes three types of difference sets: the newly added metering data, the state change set, and the rule-triggered state change set, comprising:

[0015] By comparing the physical timestamp intervals [T1, T2] and the corresponding logical sequence number ranges [S1, S2] between two state snapshots, initial data records that satisfy the condition that the timestamp belongs to the physical timestamp interval [T1, T2] or the sequence number belongs to the logical sequence number range [S1, S2] are selected, and duplicates in the initial data records are removed to obtain target data records. The data boundary between the two state snapshots is located based on the target data records, and the three types of difference sets are constructed based on the data boundaries. The physical timestamp refers to the precise time point when the data was actually generated, and the logical sequence number is a monotonically increasing identifier assigned by the system.

[0016] In one embodiment, the method further includes:

[0017] In the power load elastic response switching curve model, the throughput change rate and arrival rate change rate are weighted and summed to obtain a weighted sum result. The weighted sum result is processed through a nonlinear mapping function to obtain a data flow dynamic factor. Based on the ratio of resource utilization rate to the maximum resource utilization rate threshold in the key indicator information, a normalized resource margin is determined. Based on the resource margin, exponential scaling is performed to obtain a resource response factor. Based on the data flow dynamic factor and the resource response factor, a switching response value is obtained. Based on the switching response value, combined with the preset basic time window parameters and the maximum batch processing task capacity parameters, the optimal switching parameters are determined.

[0018] In one embodiment, the method further includes:

[0019] After the gradual task migration is completed, a final snapshot is generated for the new processing mode. The final snapshot includes the complete computation state, the latest window aggregation data, physical timestamps, and logical sequence numbers. The consistency of all operator migrations is confirmed through a two-phase commit protocol. The two-phase commit protocol includes first collecting the readiness status reports of each operator, and then sending a final commit signal after all are confirmed. If the migration is interrupted, a rollback operation is performed based on the three types of difference sets and the state snapshot to restore the state to the state before the switch. At the same time, the trigger status of the high-load type rules and the low-load type rules is updated, and a cooldown time parameter is set to prevent the processing mode switching operation from being triggered again within a preset time. After the switch is completed, the internal metadata is updated and the trigger counters of the high-load type and the low-load type are reset.

[0020] Secondly, this application also provides a device for switching between streaming and batch processing modes for metering data, comprising:

[0021] The rule matching module is used to obtain key indicator information of newly added measurement data, and to perform rule matching with high-load type rules or low-load type rules through a forward linking algorithm based on the key indicator information and a preset set of rule conditions to obtain the rule matching result; each condition in the preset set of rule conditions includes indicator name, comparison operator, threshold, duration and trigger type attribute, wherein the trigger type attribute includes high-load type and low-load type.

[0022] The direction determination module is used to count the number of triggers corresponding to the high-load type rule and the low-load type rule within a preset sliding time window according to the rule matching result, and determine the switching direction between stream processing mode and batch processing mode according to the number of triggers.

[0023] The set construction module is used to create a state snapshot of the current processing mode, obtain a state change set and a rule-triggered state change set based on the state snapshot, and construct three types of difference sets containing the newly added metering data, the state change set, and the rule-triggered state change set;

[0024] The task migration module is used to perform progressive task migration based on the three types of difference sets, wherein when the switching direction is from the stream processing mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0026] The system acquires key indicator information of newly added metering data. Based on this key indicator information and a preset set of rule conditions, it performs rule matching with high-load or low-load type rules using a forward linking algorithm to obtain the rule matching result. Each condition in the preset set of rule conditions includes an indicator name, comparison operator, threshold, duration, and trigger type attribute, where the trigger type attribute includes high-load type and low-load type. Based on the rule matching result, the system counts the number of triggers corresponding to the high-load type rules and the low-load type rules within a preset sliding time window, and determines the switching direction between stream processing mode and batch processing mode based on the number of triggers. The system creates a state snapshot of the current processing mode, and based on the state snapshot, obtains a state change set and a rule trigger state change set, and constructs three types of difference sets containing the newly added metering data, the state change set, and the rule trigger state change set. Based on the three types of difference sets, the system performs a progressive task migration, wherein when the switching direction is from the stream processing mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0028] The system acquires key indicator information of newly added metering data. Based on this key indicator information and a preset set of rule conditions, it performs rule matching with high-load or low-load type rules using a forward linking algorithm to obtain the rule matching result. Each condition in the preset set of rule conditions includes an indicator name, comparison operator, threshold, duration, and trigger type attribute, where the trigger type attribute includes high-load type and low-load type. Based on the rule matching result, the system counts the number of triggers corresponding to the high-load type rules and the low-load type rules within a preset sliding time window, and determines the switching direction between stream processing mode and batch processing mode based on the number of triggers. The system creates a state snapshot of the current processing mode, and based on the state snapshot, obtains a state change set and a rule trigger state change set, and constructs three types of difference sets containing the newly added metering data, the state change set, and the rule trigger state change set. Based on the three types of difference sets, the system performs a progressive task migration, wherein when the switching direction is from the stream processing mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0029] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0030] The system acquires key indicator information of newly added metering data. Based on this key indicator information and a preset set of rule conditions, it performs rule matching with high-load or low-load type rules using a forward linking algorithm to obtain the rule matching result. Each condition in the preset set of rule conditions includes an indicator name, comparison operator, threshold, duration, and trigger type attribute, where the trigger type attribute includes high-load type and low-load type. Based on the rule matching result, the system counts the number of triggers corresponding to the high-load type rules and the low-load type rules within a preset sliding time window, and determines the switching direction between stream processing mode and batch processing mode based on the number of triggers. The system creates a state snapshot of the current processing mode, and based on the state snapshot, obtains a state change set and a rule trigger state change set, and constructs three types of difference sets containing the newly added metering data, the state change set, and the rule trigger state change set. Based on the three types of difference sets, the system performs a progressive task migration, wherein when the switching direction is from the stream processing mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0031] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for switching between streaming and batch processing modes for metering data collects key indicator information of newly added metering data and performs rule matching based on a preset set of rule conditions using a forward linking algorithm. Then, based on the rule matching results, it counts the trigger counts for high-load and low-load rules within a preset sliding time window and determines the switching direction between streaming and batch processing modes based on the trigger counts. Next, it creates a state snapshot of the current processing mode and constructs three types of difference sets. Finally, it performs a progressive task migration based on these three types of difference sets. This application achieves intelligent dynamic switching between streaming and batch processing modes by intelligently sensing the load characteristics of metering data. This solves the technical problems of low resource utilization efficiency and poor adaptability in traditional metering data processing systems under large load fluctuations. It effectively improves the system's processing efficiency and reliability of metering data, as well as its resource utilization efficiency. It also enhances the system's adaptability to load changes, ensures data processing consistency and accuracy, and improves the system's flexibility, configurability, and adaptability to different business scenarios. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is an application environment diagram of a method for switching streaming batch processing modes for metering data in one embodiment.

[0034] Figure 2 This is a flowchart illustrating a method for switching between streaming and batch processing modes for metering data in one embodiment.

[0035] Figure 3 This is a flowchart illustrating the rule matching steps in one embodiment;

[0036] Figure 4 This is a flowchart illustrating the two-phase commit protocol steps in one embodiment;

[0037] Figure 5 This is a flowchart illustrating a specific embodiment of a method for switching between streaming and batch processing modes for metering data.

[0038] Figure 6 This is a structural block diagram of a streaming batch processing mode switching device for metering data in one embodiment.

[0039] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] The method for switching between streaming and batch processing modes for metering data provided in this application can be applied to, for example... Figure 1 The application environment shown illustrates this. In this environment, the terminal can communicate with the server via a network. The data storage system can store the data that the server needs to process. The data storage system can be integrated onto the server or located on the cloud or other network servers. In such... Figure 1 In the application environment shown, the terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0042] In one embodiment, such as Figure 2 As shown, a method for switching between streaming and batch processing modes for metering data is provided. This method can be applied to... Figure 1 In the terminal, the method may include the following steps:

[0043] Step S201: Obtain key indicator information of newly added measurement data; based on the key indicator information and the preset set of rule conditions, perform rule matching with high-load type rules or low-load type rules through the forward link algorithm to obtain the rule matching result.

[0044] Key metrics include throughput, arrival rate, and resource utilization of metering data. Throughput is the number of metering data records that the system can process per second, expressed as records per second. Arrival rate is the number of metering data records entering the system per second, expressed as records per second. Resource utilization is the percentage of current system resources in use, including CPU (Central Processing Unit) utilization and memory utilization, expressed as a percentage (%).

[0045] Each condition in the preset set of rules includes an indicator name, comparison operator, threshold, duration, and trigger type attribute, where the trigger type attribute includes high load type (…). Type) and low-load type ( (Type). The indicator name is used to identify the specific performance indicator being monitored, corresponding to the indicator array field of the collected key indicator information; comparison operators include >, >=, <, <= and ==, etc., used to construct the comparison relationship between the indicator and the threshold; the threshold represents the critical value for triggering the rule, which can be adjusted according to the system configuration and actual load characteristics; the duration defines the minimum duration required for the indicator to meet the conditions, preventing instantaneous fluctuations from triggering switching.

[0046] As an example, the process of obtaining key indicator information for newly added measurement data is as follows:

[0047] (1-1) Embed monitoring probes in the stream processing engine to continuously collect key metrics information including throughput, arrival rate and resource utilization during the metering data processing process.

[0048] As one implementation of this application, in step S201, the key metrics data of the measurement data can also be collected by calling the built-in Metrics API of the stream processing engine (such as Apache Flink or Spark Streaming). This method utilizes the engine's built-in monitoring interface to directly obtain registered metrics (such as throughput, arrival rate, CPU and memory utilization), without the need for additional probe embedding, and has low intrusiveness. After obtaining the data, it is also encapsulated and asynchronously transmitted according to steps (1-2) to (1-4).

[0049] (1-2) Convert the collected key indicator data into a structured JSON format, which includes a timestamp field, a data source identifier field, and an indicator array field. The timestamp field is a timestamp in ISO standard format, accurate to the millisecond level. The data source identifier field is a unique identifier of the source of the indicator data. The indicator array field contains an array of multiple indicator objects.

[0050] (1-3) In the indicator array field, each indicator object contains three subfields: name, value, and unit. The name field directly corresponds to the indicator name in the rule condition set; the name field is the unique identifier of the indicator, such as throughput, arrival rate, and CPU utilization; the value field is the specific value of the indicator; and the unit field is the unit of measurement of the indicator, such as records / second%.

[0051] (1-4) Pass the encapsulated JSON data to the rule matching module via an asynchronous message queue.

[0052] (1-5) The forward linking algorithm is used to perform rule matching on the received key indicator information. This algorithm treats the indicator information as facts and the rule conditions as rules, and judges whether the rule conditions are met through forward reasoning. For example, the forward linking algorithm is implemented using a Rete network to construct a rule network composed of root nodes, type nodes, pattern nodes and terminal nodes, thereby improving the rule matching efficiency.

[0053] Step S202: Based on the rule matching results, count the number of triggers for high-load type rules and low-load type rules within a preset sliding time window, and determine the switching direction between stream processing mode and batch processing mode based on the number of triggers.

[0054] As an example, the process of triggering the count is as follows:

[0055] (2-1) Within the preset sliding time window (default is 60 seconds), respectively, the statistics are calculated. Type rules and The number of times a type rule is triggered; for example, the sliding time window is implemented using a two-pointer technique, which achieves a window sliding operation with a time complexity of 0(1) while maintaining a fixed window size.

[0056] (2-2) When When a rule is triggered N times consecutively within a sliding time window (default N=5), the switching direction is determined to be from stream processing mode to batch processing mode. For example, the continuous trigger determination uses a state machine model, which can accurately identify the continuity and discontinuity of rule triggering. Optionally, the system supports setting trigger compensation parameters, allowing for a maximum of K (default K=1) instances of unmet conditions to be tolerated in N consecutive trigger determinations, thereby improving the fault tolerance of the determination.

[0057] (2-3) When When a type rule is triggered M times consecutively within a sliding time window (default M=3), the switching direction is determined to be a switch from batch processing mode to stream processing mode. For example, Trigger threshold M is less than The trigger threshold N forms a hysteresis loop to prevent frequent system switching under critical load conditions.

[0058] Step S203: Create a state snapshot of the current processing mode. Based on the state snapshot, obtain the state change set and the rule-triggered state change set, and construct three types of difference sets containing newly added metering data, the state change set, and the rule-triggered state change set.

[0059] As an example, the process of creating a state snapshot of the current processing mode and constructing three sets of differences is as follows:

[0060] (3-1) Create a stream processing state snapshot, which records the intermediate results and window aggregation data of the current operator. Intermediate results refer to the calculation state of metering data that has undergone partial processing but is not yet complete, such as intermediate values ​​in power load calculations that have completed data standardization but not time-series aggregation. Window aggregation data refers to the data successfully matched within the sliding time window. and The cumulative result of the number of rule triggers, for example, recording the number of times the rule has been triggered in the last 60 seconds. The rule was triggered 4 times. The rule was triggered once.

[0061] Each snapshot is equipped with a physical timestamp and a logical version number for precise location and backtracking. When a switching processing mode is required, the system immediately triggers a special snapshot, ensuring that the snapshot's time point is as close as possible to the switching decision point. For example, the snapshot is completed within 10 milliseconds after it is determined that the switch from stream processing to batch processing needs to be completed, keeping the incremental data volume to a minimum.

[0062] (3-2) Construct three types of difference sets to accurately locate data boundaries between snapshots based on physical timestamps and logical sequence numbers. The physical timestamp refers to the precise time point when the data was actually generated, such as "01-15T14:32:56.789Z". The logical sequence number is a monotonically increasing identifier assigned by the system, such as an integer continuously increasing from 1000000. Compare the physical timestamp range [T1, T2] and the corresponding logical sequence number range [S1, S2] between two snapshots, filter out data records that satisfy either a timestamp belonging to [T1, T2] or a sequence number belonging to [S1, S2], and remove duplicates to ensure consistent processing order.

[0063] (3-3) The three types of difference sets include: newly added metering data, including all raw indicator data generated between two snapshot time points, such as 1000 new meter readings added during the time period from T1 to T2; state change sets, recording changes in key indicators such as throughput, arrival rate, and resource utilization, such as throughput changing from 50,000 records / second to 65,000 records / second, and CPU utilization increasing from 60% to 75%; and rule-triggered state change sets, including... and The rule's trigger count increment within the sliding time window, for example The rule trigger count increased from 3 to 5. The rule trigger count remains at 0.

[0064] Step S204: Perform progressive task migration based on three types of difference sets. When the switching direction is from streaming mode to batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0065] As an example, the process of performing incremental task migration based on three sets of differences is as follows:

[0066] (4-1) Based on the calculated switching parameters and The migration task was divided into multiple batches. Determine the number of tasks to migrate in each batch, for example when When the threshold is 5000, a maximum of 5000 records can be migrated per batch. According to... Determine the batch interval time, for example when At 300 milliseconds, a new batch migration is initiated every 300 milliseconds. For example, high-priority data in newly added metering datasets, such as data from key power load monitoring points, are migrated first. After migration is complete, a final snapshot is generated for the new processing mode, recording the complete calculation status and the latest timestamp and sequence number.

[0067] (4-2) Collect the readiness status reports of each operator (such as filtering, aggregation, and correlation operators). Each operator needs to complete local state saving and confirm that it can resume operation in the new processing mode. When all operators are confirmed to be ready, the system sends a final commit signal to complete the switch. For example, when switching from stream processing to batch processing, the system first confirms that all 10 data processing operators have completed state saving, and then sends a commit signal to activate the batch processing mode.

[0068] (4-3) If an interruption occurs during the migration process, a rollback operation is performed based on the three types of difference sets and the last available snapshot to restore the state to the state before the switch. At the same time, the rule triggering state is updated to ensure system consistency. For example, a cooldown time parameter is set (typically 120 seconds) to prevent the mode switch from being triggered again during this period. After the switch is completed, the internal metadata is updated to record the current active processing mode (stream processing mode or batch processing mode) and reset. and Trigger the counter to prepare for the next potential switchover.

[0069] In this embodiment, key indicator information of newly added metering data is collected, and rule matching is performed using a forward linking algorithm based on a preset set of rule conditions. Then, based on the rule matching results, the trigger counts for high-load and low-load rules are counted within a preset sliding time window, and the switching direction between stream processing and batch processing modes is determined based on the trigger counts. Next, a state snapshot of the current processing mode is created, and three types of difference sets are constructed. Finally, a progressive task migration is performed based on these three difference sets. This application achieves intelligent dynamic switching between stream processing and batch processing modes by intelligently sensing the load characteristics of metering data. This solves the technical problems of low resource utilization efficiency and poor adaptability in traditional metering data processing systems under large load fluctuations. It effectively improves the system's processing efficiency and reliability of metering data, as well as its resource utilization efficiency. It also enhances the system's adaptability to load changes, ensures data processing consistency and accuracy, and improves the system's flexibility, configurability, and adaptability to different business scenarios.

[0070] In one embodiment, such as Figure 3 As shown, in step S202 above, determining the switching direction between stream processing mode and batch processing mode based on the number of triggers may include the following steps:

[0071] If a high-load type rule triggers a certain number of times within a preset sliding time window and reaches the high-load trigger threshold, the process switches from stream processing mode to batch processing mode; if a low-load type rule triggers a certain number of times within a preset sliding time window and reaches the low-load trigger threshold, the process switches from batch processing mode to stream processing mode.

[0072] The high-load trigger threshold N and the low-load trigger threshold M are not fixed, but are stored in an external system configuration file (such as a properties file or a YAML file). The system loads these thresholds at startup, and supports, for example, a hot reload mechanism, allowing operators to modify and apply new thresholds without restarting the system, thereby enabling dynamic adjustment of switching sensitivity to adapt to the business load characteristics at different times.

[0073] As an example, the terminal counts the data within a preset sliding time window (default is 60 seconds). Type rules and The number of times a type rule is triggered; when When a type rule is triggered N times consecutively within a sliding time window (default N=5), the switching direction is determined to be from stream processing mode to batch processing mode.

[0074] when When a type rule is triggered M times consecutively within a sliding time window (default M=3), the switching direction is determined to be from batch processing mode to stream processing mode.

[0075] In one embodiment, the state snapshot includes the intermediate results of the current operator, window aggregation data, physical timestamp, and logical sequence number. The intermediate results refer to the calculation state where the measurement data has been partially processed but not yet completed. The window aggregation data refers to the cumulative result of the number of triggers of successfully matched high-load and low-load type rules within a preset sliding time window. In step S203 above, obtaining the state change set and the rule trigger state change set based on the state snapshot may include the following steps:

[0076] Based on the changes in key indicator information obtained from the intermediate results, the changes are summarized to obtain a set of status changes; the window aggregated data is summarized to obtain a set of rule-triggered status changes.

[0077] The changes in key indicators include changes in key indicators such as throughput, arrival rate, and resource utilization of newly added metering data; window aggregation data refers to the data of successfully matched data within a sliding time window. Type rules and The cumulative result of the number of times the type rule is triggered.

[0078] As an example, the terminal obtains the changes in key indicator information based on the intermediate results of the current operator, summarizes the changes to obtain a set of state changes, and summarizes the window aggregation data of the current operator to obtain a set of rule-triggered state changes.

[0079] In one embodiment, step S203 above, constructing three types of difference sets including newly added metering data, a set of status changes, and a set of rule-triggered status changes, may include the following steps:

[0080] By comparing the physical timestamp interval [T1, T2] and the corresponding logical sequence number range [S1, S2] between two state snapshots, initial data records that satisfy the condition that the timestamp belongs to the physical timestamp interval [T1, T2] or the sequence number belongs to the logical sequence number range [S1, S2] are selected, and duplicates in the initial data records are removed to obtain the target data records. The data boundary between the two state snapshots is located based on the target data records, and three types of difference sets are constructed based on the data boundaries. The physical timestamp refers to the precise time point when the data was actually generated, and the logical sequence number is a monotonically increasing identifier assigned by the system.

[0081] As an example, the terminal compares the physical timestamp interval [T1, T2] and the corresponding logical sequence number range [S1, S2] between two state snapshots, filters out the initial data records that satisfy the condition that the timestamp belongs to [T1, T2] or the sequence number belongs to [S1, S2], and removes duplicates in the initial data records to ensure the consistency of the processing order, thus obtaining the target data record; then, based on the target data record, it locates the data boundary between the two state snapshots, and constructs three types of difference sets based on the data boundary, including newly added metering data, state change sets, and rule-triggered state change sets.

[0082] In one embodiment, the method of this application further includes the following steps:

[0083] In the power load elastic response switching curve model, the throughput change rate and arrival rate change rate are weighted and summed to obtain a weighted sum result. The weighted sum result is then processed through a nonlinear mapping function to obtain the data flow dynamic factor. Based on the ratio of resource utilization rate to the maximum resource utilization rate threshold in the key indicator information, the normalized resource margin is determined. Based on the resource margin, exponential scaling is performed to obtain the resource response factor. Based on the data flow dynamic factor and the resource response factor, the switching response value is obtained. Based on the switching response value, combined with the preset basic time window parameters and the maximum batch processing task capacity parameters, the optimal switching parameters are determined.

[0084] The power load elastic response switching curve model is defined as follows:

[0085]

[0086] The meanings and example values ​​of each parameter in this model are as follows:

[0087] SR(t) is the switching response coefficient of the system at time t, and its value ranges from [0,1]. For example, SR(t)=0.3 indicates a low response coefficient and a slow switching, while SR(t)=0.8 indicates a high response coefficient and a fast switching. The basic switching rate coefficient controls the overall response sensitivity. The typical value is 0.8. In scenarios with high availability requirements, it can be set to 0.9, and in scenarios with high stability requirements, it can be set to 0.7. ΔA(t) is the throughput change rate, calculated based on the throughput data collected in step S201; ΔA(t) is the arrival rate change rate, calculated based on the arrival rate data collected in step S201. To determine the current resource utilization rate, take For example, if the CPU utilization is 75% and the memory utilization is 68%, then U(t) = 0.75; Sets the maximum resource utilization threshold for the system to prevent system overload. The typical value is 0.9. For high-performance servers, it can be set to 0.95, and for older equipment, it can be set to 0.85. and These are the characteristic weights for power data, representing the sensitivity to changes in throughput and arrival rate, respectively. Typical values ​​are 0.6 and 0.4. For power metering data, since the computational complexity is relatively fixed, they can be set to... =0.65, =0.35; This parameter controls the influence of resource utilization on switching response. A typical value is 2; the higher the value, the more significant the suppression effect when approaching resource limits. In high-load scenarios, it can be set to 3 to allow the system to adjust load earlier.

[0088] Based on the above model, calculate the optimal switching parameters:

[0089] Switch time windows:

[0090] In the above formula, This is the base time window parameter, typically 200 milliseconds;

[0091] Number of tasks per batch:

[0092] In the above formula, This is the maximum batch processing capacity parameter, typically 10,000 records.

[0093] Based on the calculated parameters, the operators of the measurement data are divided:

[0094] The number of jobs contained in each batch shall not exceed For example, when When the value is 5000, the system can process a maximum of 5000 records in a batch at a time.

[0095] Batch interval time follows For example, when =At 300 milliseconds, the system starts a new batch every 300 milliseconds.

[0096] In practical applications, when the grid load increases from a normal level, such as the throughput increasing from 30,000 packets / second to 60,000 packets / second within 5 minutes, and the arrival rate increasing from 40,000 packets / second to 75,000 packets / second, the SR(t) value will increase from 0.35 to 0.75. The system will automatically adjust... From approximately 570 milliseconds to approximately 270 milliseconds, while... Increased from 3,500 to 7,500 to speed up batch processing to cope with increased load.

[0097] In one embodiment, the method of this application further includes the following steps:

[0098] After the incremental task migration is completed, a final snapshot is generated for the new processing mode. The final snapshot contains the complete computation state, the latest window aggregation data, physical timestamps, and logical sequence numbers. The consistency of all operator migrations is confirmed through a two-phase commit protocol. The two-phase commit protocol includes first collecting the readiness status reports of each operator, and then sending a final commit signal after all are confirmed. If the migration is interrupted, a rollback operation is performed based on three sets of differences and the status snapshot to restore the state to the state before the switch. At the same time, the trigger status of high-load type rules and low-load type rules is updated, and a cooldown time parameter is set to prevent the processing mode switching operation from being triggered again within a preset time. After the switch is completed, the internal metadata is updated and the trigger counters for high-load type and low-load type are reset.

[0099] As an example, such as Figure 4 As shown, the switching parameters are based on calculations. and The migration task was divided into multiple batches, according to... Determine the number of tasks to migrate in each batch, for example when When the threshold is 5000, a maximum of 5000 records can be migrated per batch; according to... Determine the batch interval time, for example when At 300 milliseconds, a new batch migration is initiated every 300 milliseconds. For example, high-priority data in newly added metering datasets, such as data from key power load monitoring points, are migrated first. After migration, a final snapshot is generated for the new processing mode, recording the complete calculation status and the latest timestamp and sequence number. Readiness status reports for each operator (such as filtering, aggregation, and correlation operators) are collected. Each operator needs to complete local state saving and confirm that it can resume operation in the new processing mode. When all operators are confirmed to be ready, the system sends a final commit signal to complete the switch. For example, when switching from stream processing to batch processing, the system first confirms that all 10 data processing operators have completed state saving, and then sends a commit signal to activate the batch processing mode. If an interruption occurs during migration, a rollback operation is performed based on the three types of difference sets and the last available snapshot to restore the state before the switch, while updating the rule trigger state to ensure system consistency. For example, a cooldown time parameter (typically 120 seconds) is set to prevent the mode switch from being triggered again during this period. After the switch is completed, internal metadata is updated to record the currently active processing mode (stream processing or batch processing) and reset. and Trigger the counter to prepare for the next potential switchover.

[0100] In one embodiment, such as Figure 5 As shown, a method for switching between streaming and batch processing modes for metering data is provided in a specific embodiment, which includes the following steps:

[0101] Step S501: Obtain key indicator information of newly added metering data; based on the key indicator information and the preset set of rule conditions, perform rule matching with high-load type rules or low-load type rules through the forward link algorithm to obtain the rule matching result.

[0102] Step S502: Based on the rule matching results, count the number of triggers for high-load and low-load rules within a preset sliding time window; if the number of triggers for high-load rules within the preset sliding time window reaches the high-load trigger threshold, switch from stream processing mode to batch processing mode; if the number of triggers for low-load rules within the preset sliding time window reaches the low-load trigger threshold, switch from batch processing mode to stream processing mode.

[0103] Step S503: Create a state snapshot of the current processing mode, obtain the changes in key indicator information based on intermediate results, summarize the changes to obtain a state change set, summarize the window aggregated data to obtain a rule-triggered state change set, and construct three types of difference sets including newly added measurement data, state change set, and rule-triggered state change set.

[0104] Step S504: Perform incremental task migration based on three types of difference sets. After the incremental task migration is completed, generate a final snapshot for the new processing mode. The final snapshot contains the complete computing state as well as the latest window aggregation data, physical timestamp and logical sequence number.

[0105] Step S505: Confirm the consistency of all operator migrations through a two-phase commit protocol. The two-phase commit protocol includes first collecting the readiness status reports of each operator, and sending a final commit signal after all are confirmed. If the migration is interrupted, perform a rollback operation based on the three types of difference sets and the status snapshot to restore the state before the switch. At the same time, update the trigger status of the high-load type rules and the low-load type rules, and set a cooldown time parameter to prevent the processing mode switching operation from being triggered again within a preset time. After the switch is completed, update the internal metadata and reset the trigger counters for the high-load type and the low-load type.

[0106] The beneficial effects of the above embodiments are as follows:

[0107] (1) The dynamic switching method of this application realizes automatic switching between stream processing and batch processing modes by intelligently sensing the load characteristics of metering data, effectively improving the system resource utilization efficiency, solving the problem of resource waste or insufficient performance under the fixed processing mode, and improving the overall system operating efficiency.

[0108] (2) Based on the power load elastic response switching curve model SR(t), this application dynamically calculates the optimal switching parameters, which can accurately respond to the load fluctuation of the power system according to the throughput rate change rate, arrival rate change rate and resource utilization rate. This solves the problem of poor adaptability of the traditional fixed parameter switching method and improves the system's adaptability to load changes.

[0109] (3) This application adopts a progressive task migration and two-stage submission protocol mechanism, combined with three types of difference sets that are precisely constructed, to ensure data consistency during the processing mode switching process, solve the risk of data loss or duplicate processing during the mode switching process, and improve the reliability and accuracy of metrological data processing.

[0110] (4) This application supports Type rules and The two types of rules can be combined using AND / OR / NOT logic, and the trigger threshold can be dynamically adjusted through configuration files. This solves the problem of insufficient system flexibility caused by fixed rule conditions and improves the system's adaptability to different business scenarios.

[0111] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0112] Based on the same inventive concept, this application also provides a device for switching the batch processing mode of metering data to implement the above-mentioned method for switching the batch processing mode of metering data. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the device for switching the batch processing mode of metering data provided below can be found in the limitations of the method for switching the batch processing mode of metering data described above, and will not be repeated here.

[0113] In one exemplary embodiment, such as Figure 6 As shown, a streaming batch processing mode switching device for metering data is provided, which may include:

[0114] The rule matching module 601 is used to obtain key indicator information of newly added measurement data, and to perform rule matching with high load type rules or low load type rules through a forward linking algorithm based on the key indicator information and a preset set of rule conditions to obtain the rule matching result. Each condition in the preset set of rule conditions includes indicator name, comparison operator, threshold, duration and trigger type attribute, where the trigger type attribute includes high load type and low load type.

[0115] The direction determination module 602 is used to count the number of triggers corresponding to high-load type rules and low-load type rules within a preset sliding time window based on the rule matching results, and determine the switching direction between stream processing mode and batch processing mode based on the number of triggers.

[0116] The set construction module 603 is used to create a state snapshot of the current processing mode. Based on the state snapshot, a state change set and a rule-triggered state change set are obtained, and three types of difference sets are constructed, including newly added metering data, state change set, and rule-triggered state change set.

[0117] The task migration module 604 is used to perform progressive task migration based on three types of difference sets. When the switching direction is from stream processing mode to batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

[0118] In one embodiment, the direction determination module 602 is further configured to switch from stream processing mode to batch processing mode when the number of triggers of a high-load type rule within a preset sliding time window reaches a high-load trigger threshold; and to switch from batch processing mode to stream processing mode when the number of triggers of a low-load type rule within a preset sliding time window reaches a low-load trigger threshold.

[0119] In one embodiment, the set construction module 603 is further configured to obtain the changes in key indicator information based on intermediate results, summarize the changes to obtain a state change set, and summarize the window aggregated data to obtain a rule-triggered state change set.

[0120] In one embodiment, the set construction module 603 is further configured to compare the physical timestamp interval [T1, T2] and the corresponding logical sequence number range [S1, S2] between two state snapshots, filter out initial data records that satisfy the condition that the timestamp belongs to the physical timestamp interval [T1, T2] or the sequence number belongs to the logical sequence number range [S1, S2], and remove duplicates in the initial data records to obtain target data records; locate the data boundary between the two state snapshots based on the target data records, and construct three types of difference sets based on the data boundaries; wherein the physical timestamp refers to the precise time point when the data was actually generated, and the logical sequence number is a monotonically increasing identifier assigned by the system.

[0121] In one embodiment, the device may further include: a parameter determination module, configured to: perform a weighted summation of the throughput change rate and the arrival rate change rate in the power load elastic response switching curve model to obtain a weighted summation result; process the weighted summation result through a nonlinear mapping function to obtain a data flow dynamic factor; determine a normalized resource margin based on the ratio of resource utilization rate to the maximum resource utilization rate threshold in key indicator information; perform exponential scaling processing based on the resource margin to obtain a resource response factor; obtain a switching response value based on the data flow dynamic factor and the resource response factor; and determine the optimal switching parameters based on the switching response value, combined with preset basic time window parameters and maximum batch processing task capacity parameters.

[0122] In one embodiment, the apparatus may further include: a rollback operation module, configured to generate a final snapshot for the new processing mode after the gradual task migration is completed, the final snapshot containing the complete computation state and the latest window aggregation data, physical timestamp, and logical sequence number; confirm the consistency of all operator migrations through a two-phase commit protocol, the two-phase commit protocol including first collecting the readiness status reports of each operator, sending a final commit signal after all are confirmed, and if the migration is interrupted, performing a rollback operation based on three types of difference sets and the state snapshot to restore the state before the switch, while updating the trigger status of high-load type rules and low-load type rules, and setting a cooldown time parameter to prevent the processing mode switching operation from being triggered again within a preset time; after the switch is completed, updating the internal metadata and resetting the trigger counters for high-load type and low-load type.

[0123] Each module in the aforementioned batch processing mode switching device for metering data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0124] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for switching between streaming and batch processing modes for metering data. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0125] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0126] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0127] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0128] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0130] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0131] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0132] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for switching between batch processing modes for metering data, characterized in that, The method includes: The system acquires key indicator information of newly added measurement data, and performs rule matching with high-load type rules or low-load type rules through a forward linking algorithm based on the key indicator information and a preset set of rule conditions to obtain the rule matching result. Each condition in the preset set of rule conditions includes indicator name, comparison operator, threshold, duration and trigger type attribute, wherein the trigger type attribute includes high-load type and low-load type. Based on the rule matching results, the number of triggers corresponding to the high-load type rule and the low-load type rule are counted within a preset sliding time window, and the switching direction between stream processing mode and batch processing mode is determined based on the number of triggers. Create a state snapshot of the current processing mode. Based on the state snapshot, obtain a state change set and a rule-triggered state change set, and construct three types of difference sets containing the newly added metering data, the state change set, and the rule-triggered state change set. A progressive task migration is performed based on the three sets of differences, wherein when the switching direction is from the streaming mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

2. The method according to claim 1, characterized in that, The step of determining the switching direction between stream processing mode and batch processing mode based on the number of triggers includes: If the number of times the high-load type rule is triggered within the preset sliding time window reaches the high-load trigger threshold, the stream processing mode is switched to the batch processing mode. If the number of times the low-load type rule is triggered within the preset sliding time window reaches the low-load trigger threshold, the batch processing mode is switched to the stream processing mode.

3. The method according to claim 2, characterized in that, The state snapshot includes the intermediate results of the current operator, window aggregation data, physical timestamp, and logical sequence number. The intermediate results refer to the calculation state where the measurement data has been partially processed but not yet completed. The window aggregation data refers to the cumulative result of the number of triggers of the successfully matched high-load type rule and the low-load type rule within the preset sliding time window. The step of obtaining the state change set and the rule-triggered state change set based on the state snapshot includes: Based on the intermediate results, the changes in the key indicator information are obtained, and the changes are summarized to obtain the state change set; The aggregated data in the window is summarized to obtain the set of rule-triggered state changes.

4. The method according to claim 3, characterized in that, The construction includes three types of difference sets: the newly added metering data, the status change set, and the rule-triggered status change set. By comparing the physical timestamp interval [T1, T2] and the corresponding logical sequence number range [S1, S2] between two state snapshots, initial data records that satisfy the condition that the timestamp belongs to the physical timestamp interval [T1, T2] or the sequence number belongs to the logical sequence number range [S1, S2] are selected, and duplicate items in the initial data records are removed to obtain the target data record; The data boundary between the two state snapshots is located based on the target data record, and the three types of difference sets are constructed based on the data boundary; The physical timestamp refers to the precise time point at which the data was actually generated, and the logical sequence number is a monotonically increasing identifier assigned by the system.

5. The method according to claim 1, characterized in that, The method further includes: In the power load elastic response switching curve model, the throughput change rate and arrival rate change rate are weighted and summed to obtain a weighted sum result. The weighted sum result is processed through a nonlinear mapping function to obtain a data flow dynamic factor. Based on the ratio of resource utilization rate to the maximum resource utilization rate threshold in the key indicator information, a normalized resource margin is determined. Based on the resource margin, an exponential scaling process is performed to obtain a resource response factor. Based on the data flow dynamic factor and the resource response factor, a switching response value is obtained; based on the switching response value, combined with the preset basic time window parameter and the maximum batch processing task capacity parameter, the optimal switching parameters are determined.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: After the gradual task migration is completed, a final snapshot is generated for the new processing mode. The final snapshot contains the complete computing state as well as the latest window aggregation data, physical timestamps and logical sequence numbers. The consistency of all operator migrations is confirmed through a two-phase commit protocol. The two-phase commit protocol includes first collecting the readiness status reports of each operator, and then sending a final commit signal after all are confirmed. If the migration is interrupted, a rollback operation is performed based on the three types of difference sets and the status snapshot to restore the state to the state before the switch. At the same time, the trigger status of the high-load type rule and the low-load type rule is updated, and a cooldown time parameter is set to prevent the processing mode switching operation from being triggered again within a preset time. After the switch is completed, the internal metadata is updated and the trigger counters of the high-load type and the low-load type are reset.

7. A device for switching between streaming and batch processing modes for metering data, characterized in that, The device includes: The rule matching module is used to obtain key indicator information of newly added measurement data, and to perform rule matching with high-load type rules or low-load type rules through a forward linking algorithm based on the key indicator information and a preset set of rule conditions to obtain the rule matching result; each condition in the preset set of rule conditions includes indicator name, comparison operator, threshold, duration and trigger type attribute, wherein the trigger type attribute includes high-load type and low-load type. The direction determination module is used to count the number of triggers corresponding to the high-load type rule and the low-load type rule within a preset sliding time window according to the rule matching result, and determine the switching direction between stream processing mode and batch processing mode according to the number of triggers. The set construction module is used to create a state snapshot of the current processing mode, obtain a state change set and a rule-triggered state change set based on the state snapshot, and construct three types of difference sets containing the newly added metering data, the state change set, and the rule-triggered state change set; The task migration module is used to perform progressive task migration based on the three types of difference sets, wherein when the switching direction is from the stream processing mode to the batch processing mode, the progressive task migration is performed based on the optimal switching parameters determined by the power load elastic response switching curve model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Stream processing and batch processing switching method and switching device

    CN116841753A

  • Distributed big data real-time stream processing method and system

    CN120069501A