A flow processing job scaling scheduling method based on priority state migration
By employing a priority state transition-based scaling and scaling method for stream processing jobs, and utilizing a global controller and control signal mechanism, combined with a progressive migration strategy and message queue technology, the blocking and resource usage issues during the state transition process of stream processing systems are resolved, achieving efficient and interference-free system scaling and scaling.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2022-07-04
- Publication Date
- 2026-06-09
Smart Images
Figure CN115168006B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of streaming job scheduling and big data processing, and in particular to a method for scaling up and down streaming processing jobs based on priority state transition. Background Technology
[0002] Cloud computing, as a new type of information infrastructure service supporting the operation of numerous industry applications, is receiving increasing attention from the industry. With the continuous evolution of cloud computing technology, cloud-native technologies are being gradually promoted and adopted due to their lightweight, fine-grained, and highly elastic characteristics. Unlike traditional cloud computing, cloud-native reduces resource allocation and task scheduling to the container level, greatly improving the computing efficiency of cloud computing and is considered a key technology for next-generation cloud computing by many cloud vendors. Stream processing systems on cloud platforms are an important research and development direction for current stream processing systems.
[0003] Modern distributed stream processing systems typically employ a horizontally scalable design pattern. In this pattern, the stream processing task is represented as a directed acyclic graph (DAG), where each node represents a stream processing operator. Each operator receives data from its upstream operator, processes the data, and sends the results to downstream operators. Operators without upstream nodes are called source operators, and those without downstream nodes are called sink operators. In actual operation, distributed stream processing systems perform computation in a data-parallel manner. Specifically, an operator is mapped to multiple parallel instances. The number of instances of an operator is called its degree of parallelism.
[0004] Since stream processing operators are typically stateful and their state is partitioned among worker nodes, scaling an operator will trigger state transitions. This means that the system needs to move state data between worker nodes, a process known as state redistribution. This results in a large amount of data transmission, even across networks.
[0005] State transition in a stream processing system is an important and challenging problem. For a distributed stateful stream processing system, an ideal scaling mechanism should satisfy the following properties: (1) Low blocking during scaling: During the scaling phase, the operation of stream processing should be blocked as little as possible; (2) Low resource consumption during scaling: During the scaling phase, excessive redundant resources should not be used; (3) No interference during non-scaling: During the non-scaling phase, the execution of normal stream processing tasks should not be affected.
[0006] Currently, research on state transitions in stream processing systems can be categorized into four types: global restart, which pauses and restarts the entire task during state redistribution; local pause, which restarts a subgraph of the task topology rather than the entire job topology to reduce blocking of running tasks; redundant data streams starting a new stream processing task to temporarily take over the execution of the original task during state transitions; and pre-transition mechanisms, which add extra execution logic to stream processing tasks during non-scaling periods to distribute the system's operational pressure during scaling. However, existing work can only satisfy at most two of the above, either facing a significant increase in processing latency or high resource utilization, or significantly impacting stream processing during non-scaling phases.
[0007] To satisfy the above three properties, our invention faces the following challenges: state consistency, scaling the system without interrupting the streaming task and ensuring the consistency of the global state of the streaming task; migration efficiency, maximizing the efficiency of state data migration to minimize the performance degradation of operators caused by state migration; and interference locality, avoiding impact on system performance during non-scaling phases. Summary of the Invention
[0008] Purpose of the invention: In view of the problems and deficiencies of the existing technology, the purpose of this invention is to propose a stream processing job scaling and scaling scheduling method based on priority state transition. For the real-time scaling and state transition problems of stateful stream processing systems, this invention proposes a priority transition mechanism, which includes hotkey transition, data management, system design, etc., to improve the latency performance and throughput of the stream processing system during system resource allocation without using a large amount of resources or interfering with normal stream processing tasks.
[0009] Technical Solution: To achieve the above-mentioned objectives, the technical solution adopted by this invention is a method for scaling up and down stream processing jobs based on priority state transitions, comprising the following steps:
[0010] (1) Preparation phase: The global controller updates the task topology within the stream processing task according to the new parallelism; then, the global controller verifies the current key space C of the stream processing task with all operator instances. k ;
[0011] (2) Allocation Phase: When a stream processing task triggers a scaling operation during execution, the global controller starts a new operator instance for the stream processing task and creates new data channels for the upstream and downstream operators accordingly; then, it records the future key space F of each operator that needs to update its parallelism. k ;
[0012] (3) Migration phase: The upstream operator updates its data distribution strategy and uses a control signal-based mechanism to coordinate the state migration between operator instances. In addition, a progressive migration strategy is used to perform inter-instance state migration on operators that are being expanded or shrunk, and the state to be migrated is divided into several parts for migration, each part being called a micro-batch.
[0013] (4) Transfer and cleanup phase: Destroy instances that are no longer in use after the scaling down operation; at the same time, reclaim temporary system resources allocated during scaling up.
[0014] Furthermore, in step (1), a hierarchical data structure is used to implement state management. Based on the single-layer mapping, each key group is further divided into multiple sub-key groups. The above design can keep the regular stream processing running in a coarse-grained manner to avoid additional overhead, and perform state transitions in a fine-grained manner to reduce the impact on operator performance.
[0015] Furthermore, in step (2), the first operator instance I a Given a set of states S m Migration to the second operator instance I b The state S m It contains three state values, corresponding to keys k1, k2, and k3 respectively; after triggering the state transition, the first operator instance I... a First, the state to be migrated is sent to the second operator instance I as batch data. b This is called a send operation; and during this process, the second operator instance I... b In addition to waiting for batch data to arrive, it can also actively retrieve individual data entries, thus enabling timely processing. This is called a retrieval operation.
[0016] Furthermore, in step (3), at the beginning of each transition phase, the global controller is responsible for injecting a control signal into the source operator, thereby triggering a state transition; subsequently, the control signal can flow through the entire task topology; whenever the first operator instance I... a When a data record is received from the input data channel, the control signal is first sent to the downstream second operator instance I. b If the downstream second operator instance I b If a state transition is required, the data distribution strategy is updated. When a state transition is required, if not all control signals have been received, the system enters the alignment state; otherwise, it enters the alignment completed state. Finally, an acknowledgment message is sent to the global controller. If the global controller has received acknowledgment messages from all parallel instances of the task, it signifies the completion of a transition.
[0017] Furthermore, in step (3), during the migration of each microbatch, the global controller determines which states should be migrated based on the user-defined microbatch size; during the migration of each microbatch, the control signal contains information about all keys that need to be migrated, and each upstream instance creates a temporary routing table that determines which downstream instance to send the record to.
[0018] Furthermore, in step (4), based on the demand queue and the response queue, and using an external key-value storage service, state transfer services are provided during the migration phase; during the migration phase, each operator instance, in addition to the normal processing logic, starts two additional lightweight threads to continuously read messages from the demand queue and the response queue.
[0019] Beneficial effects: During dynamic scaling, this invention utilizes a control signal mechanism to synchronize and coordinate various operator instances, ensuring global state consistency and preventing disruption of the original stream processing task's semantics during migration. It introduces a multi-granularity state access mechanism to achieve lightweight state access and migration, improving data transmission efficiency during state migration. Furthermore, it designs a low-intrusion scaling module that is pluggable into existing systems and introduces no operations during non-scaling phases. In addition, this invention introduces a multi-stage scaling process and designs a message queue-based inter-operator communication mechanism to achieve efficient state transmission. Attached Figure Description
[0020] Figure 1 This is an overall flowchart of the method of the present invention;
[0021] Figure 2 (a)(b)(c)(d) are schematic diagrams showing the comparison results of the end-to-end latency during the scaling phase when the implementation of the method of the present invention in Fink 1.12.0 and the comparison method are run in NEXMark Q1, Q3, Q5 and Q7, respectively. Detailed Implementation
[0022] The present invention will be further illustrated below with reference to the accompanying drawings and specific examples. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0023] This invention proposes a stream processing job scaling scheduling method based on priority state transition, which solves the problems of state consistency, transition efficiency, and interference locality in stream processing state transitions. Figure 1 As shown, the complete process of this invention includes a preparation stage, an allocation stage, a migration stage, and a cleanup stage. Specific implementation methods are described below:
[0024] The preparation phase corresponds to technical solution step (1). The specific implementation is as follows: Based on a single-layer mapping, each key group is further divided into multiple sub-key groups. When an operator instance encounters a data record, if processing this record requires a state not located locally, the instance will attempt to "acquire" the sub-key group corresponding to this record, rather than the entire key group. This saves the overhead of "acquiring" states that are not immediately needed. In other words, normal stream processing can still be performed according to coarse-grained key groups, while state transitions can be performed according to fine-grained sub-key groups, reducing the impact of state transitions on the performance of stream processing jobs.
[0025] Because key groups are distributed across different operator instances, each instance needs to store metadata to know which key groups it is responsible for. Therefore, as the number of key groups increases, the overhead of metadata management and data record distribution also increases. In contrast, for subkey groups, when the system is not scaling, subkey groups with the same key group must belong to the same operator instance. Therefore, increasing the number of subkey groups does not introduce significant additional overhead to the data flow in a non-scalable state. In practice, users can choose an appropriate number of subkey groups based on the specific job's state size and the expected maximum latency during system scaling to achieve smooth state transitions.
[0026] The corresponding technical solution step (2) is as follows: The specific implementation method is: Represent the data record with key k as r. k The state value with key k is represented as v. k It is worth noting that in stateful distributed stream processing systems, there is an important fact: when partitioning among operator instances, the partitioning method for operator state is exactly the same as the partitioning method for data records. This means that when the first operator instance I... a Process a record r k At that time, it needs and only needs to access v k Furthermore, when one or more operator instances are processing data records with different keys, they do not need to access any common state. Based on this property, this invention proposes an on-demand state access mechanism that enables the scaling and data processing of a stream processing system to run simultaneously. Specifically, when the stream processing system triggers scaling, it triggers an online state reallocation: the first operator instance I... a Given a set of states S m Migration to the second operator instance I b Among them, this set of states S m There are three state values. The set of three states, each corresponding to a key k1, k2, or k3. The above process can also be expressed by the formula:
[0027]
[0028] After the state transition is triggered, the first operator instance I a First, the state to be migrated is sent to the second operator instance I as batch data. b This is called a send operation. During this process, the second operator instance I... b Besides waiting for batch data to arrive, it can also actively retrieve the required individual data entries, thus enabling timely processing. This is called a fetch operation. For example, if the second operator instance I... b In full receipt of state S m Previously, it was necessary to process a data record. It not only passively waits for the batch data to arrive, but also immediately sends data to the first operator instance I. a Actively retrieve a single data entry Compared to batch data transfer, the amount of data in a single data entry is much smaller. Therefore, retrieving a single data entry is a lightweight operation with only a small communication latency. Therefore, the second operator instance I... b It can be generated in a timely manner The calculation results, without needing to fully receive state S m It was blocked before.
[0029] In this way, the system can maintain the normal operation of stream processing operators during scaling. In this mechanism, sending batch data aims for rapid state transitions, while proactive fetching ensures that instances can quickly process data even when non-local state is required. During state transitions, stream processing latency is only affected when an operator encounters a data record that requires a non-local state to process. In this case, only the processing of that single record incurs transition overhead and is delayed due to an additional state fetch operation. In other words, the performance disruption caused by state transitions is reduced from the job and operator granularity to the message granularity, thus reducing subsequent queuing overhead. Based on this, the system can maintain the real-time performance of stream processing as much as possible during scaling.
[0030] The technical solution for the migration preparation phase is step (3). Specifically, this invention designs a mechanism based on control signals to coordinate state transitions between operator instances. First, the time period from the start to the end of a state reallocation is called the migration phase. Next, this paper introduces the concept of control signals. A control signal is a special data record generated and injected into the data stream by the global controller of the stream processing system. Specifically, at the beginning of each migration phase, the global controller is responsible for injecting a control signal into the source operator, thereby triggering a state transition. Then, this signal flows through the entire task topology like a regular data record.
[0031] Whenever an operator instance I receives a data record from its input data channel, it immediately performs the following steps: (1) First operator instance I a The second operator instance I sends the control signal downstream. b (2) If the second operator instance I b State transition is required; first operator instance I a Update its data distribution strategy; (3) If the first operator instance I a It needs to undergo state transition. If it has not yet received control signals from all input channels, it enters the alignment state; otherwise, it enters the alignment completed state. (4) After completing the above steps, the first operator instance I a Send an acknowledgment message to the global controller. If the global controller receives acknowledgments from all parallel instances of the task, then a migration phase is complete.
[0032] During state transitions, stream processing operators can continue running without explicitly blocking their input channels. The impact of state transitions on stream processing tasks is only perceived when an instance needs remote state: a lightweight operation (fetching a single state record) increases the processing latency of that record. For subsequent records, if the operator instance has already obtained the corresponding state, the processing of these records is completely unaffected by the transition overhead. In this way, the impact of state transitions can be significantly reduced, thereby avoiding sudden increases in latency and decreases in throughput when the stream processing system scales.
[0033] The technical solution step (4) corresponds to the migration phase. The specific implementation is as follows: A progressive migration strategy is designed, dividing the aforementioned states to be migrated into several micro-batches for migration. In stream processing, global state consistency usually refers to exactly-once semantics. Exact-once semantics means that all data has one and only one impact on the state; this means that in a migration phase, if the state with key k is transferred from the first operator instance I... a It was migrated to the second operator instance I b In this process, each data record entering the stream processing system has an impact on the final calculation result only once.
[0034] Let t1 be instance I of the first operator. a Or the second operator instance I b Let t2 be the moment when the control signal is first received, and let t2 be the first operator instance I. a Second operator instance I b When control signals have been received from all input channels, the alignment phase begins at t1. During this phase, the record with key k can be sent to the first operator instance I. a It may be sent to the second operator instance I.b However, only one instance can hold the state of k at any given time. In other words, only one instance can modify this state at any given time. Since each data record can only be processed once, and the state can only be borrowed by other instances after the current record has been processed, the exact-once semantics of stream processing are maintained within the time interval [t1, t2]. After t2, the alignment completion phase begins, and subsequent data records will only be sent to the second operator instance I. b Therefore, the second operator instance I b With at most one borrow operation, the state can be fetched locally and processed once for each record until the alignment phase ends, thus maintaining consistent semantics during this period.
[0035] Most distributed stream processing systems supporting exact-once semantics provide checkpointing or snapshot mechanisms, enabling the system to recover to a globally consistent state after a failure, thus achieving system fault tolerance. As a dynamic scaling mechanism, this invention inherits fault-tolerant characteristics from the underlying stream processing system. Specifically, if an existing stream processing system implements the mechanism of this invention, it will normally perform its periodic checkpointing or snapshot operations during the normal operation phase; during the aforementioned migration phase, checkpointing or snapshots are disabled, and if a system failure occurs during this period, the current scaling will also be considered a failure, and the stream processing system will recover from the most recent checkpoint or snapshot.
[0036] When stateful stream processing systems scale, even if the parallelism doesn't change significantly, it may still be necessary to migrate most of the operator states. For example, if the state is evenly distributed across all operator instances, changing the operator parallelism from 2 to 3 will cause half of the keys to change positions. In practical applications, for an operator with 128 key groups, changing its parallelism from 25 to 30 requires migrating 115 of the 128 key groups, which is almost equivalent to a full state migration. Moving all these states during a migration phase significantly degrades the overall system performance because most of the records processed by the system during the migration may be affected by state acquisition operations. As data continues to flow in, these small processing delays can gradually accumulate, eventually leading to high latency peaks.
[0037] To address this issue, this invention designs a progressive migration strategy that breaks down the aforementioned states to be migrated into several micro-batches for migration. A migration phase in a system scaling process actually consists of multiple progressive migration steps. In each progressive migration step, the global controller of the stream processing system determines which states should be migrated based on the user-defined micro-batch size. For example, a micro-batch size of 1 means that in each progressive migration step, each instance can migrate at most one key group state; therefore, a migration phase is divided into four progressive migration steps. In each progressive migration step, the control signal contains information about all the keys to be migrated in that step, while each upstream instance creates a temporary routing table to determine which downstream instance it should send the record to. In this way, in each progressive migration step, only a small portion of all the states to be migrated are affected, while most data records can be processed normally. Therefore, the system processing latency curve can be flattened instead of increasing sharply. By selecting different micro-batch sizes, users can trade off between lower latency peaks and higher migration throughput.
[0038] The corresponding technical solution step (4) for the cleanup phase is as follows: Based on two message queues, a demand queue and a response queue, an external key-value storage service is used to perform state transfer during the migration phase. In addition to the normal processing logic, each operator instance will start two lightweight threads to continuously read messages from these two message queues.
[0039] For example, let's illustrate a state from the first operator instance I. a To the second operator instance I b The transition process: (1) When the second operator instance I b When it needs to retrieve the state with the key "6", it doesn't need to know which instance the state currently belongs to; it only needs to push a message "I" to the demand queue. b -6” means “I” b "Requesting the state for key 6". (2) First operator instance I a The message is read from the message queue and stored in its key-value space C. k Find "6". Therefore, the first operator instance I a Write the state value to an external key-value store, and treat its local key "6" state as borrowed. (3) First operator instance I a Push message "I" to the response queue b "-6" indicates that the request has been completed. (4) Second operator instance I b The message is retrieved from the response queue, and then the corresponding data is read from an external key-value store.
[0040] In the above process, there are two special cases: (1) When the request message arrives at the first operator instance I a When, if the first operator instance I a Currently processing a data record with key 6, the first operator instance I will remain until it finishes processing the current record. a Only then will it respond to this request; (2) if the state with key 6 has already been handled by the first operator instance I a Pushed to external state storage, then the first operator instance I a A record requiring this state has been received. Since "6" no longer belongs to the first operator instance I... a Therefore, the first operator instance I a A similar process should be triggered to obtain this state. If the system is not operating under severe load imbalance, the above situations will only last for a short time, so neither of them will cause a significant performance degradation during system scaling.
[0041] To decouple the components in a distributed system, many existing scalable message queuing technologies can be used to implement the publish-subscribe model described above. These message queuing technologies are already common components in practical stream processing applications because they can be used as data sources and receivers for streaming data, making them naturally suitable for the current stream processing paradigm. For external key-value storage services, existing high-speed key-value storage technologies with fault tolerance guarantees (such as Redis) can be selected to ensure real-time computation.
[0042] To test the performance of this invention, experiments were conducted on a cluster containing six compute nodes, using the NEXMark benchmark suite and a key-count job as the workload. The prototype system of this invention is implemented based on Apache Flink 1.12.0. For comparison, the performance of traditional methods Native Flink and Order-Unaware was also evaluated. Figure 2 This diagram illustrates the comparison of end-to-end latency over time during the scaling phase when running NEXMark partial queries for the method of this invention and the comparative method.
[0043] NEXMark Q1 performs currency conversion operations on order data streams. For example... Figure 2 As shown in (a), the present invention does not exhibit any latency peaks throughout the process, only small-range latency fluctuations caused by system noise. This is because they do not generate state transition costs during system scaling, and scaling operations can be completed asynchronously.
[0044] NEXMark Q3 tests join query tasks. For example... Figure 2As shown in (b), the present invention exhibits no significant changes in processing latency throughout the entire process because it reduces the interference caused by state transitions to the message granularity. Thanks to its on-demand acquisition mechanism and gradual migration strategy, the peak latency of the method in this invention during scaling is an order of magnitude smaller than that of the other two methods.
[0045] NEXMark Q5 is used to test window operators with small state variables. For example... Figure 2 As shown in (c), because the state quantity is very small, the state transition cost during expansion is very low, so the recording processing latency of the method of the present invention is hardly affected.
[0046] NEXMark Q7 tests window operators with larger states, performing scrolling window and join operations on two data streams. For example... Figure 2 As shown in (d), since the method of the present invention can prioritize the migration of hotkey states, thereby obtaining a smoother latency curve, the real-time performance of the calculation is better maintained during the expansion period. Therefore, the latency peak during the system expansion period is at least an order of magnitude smaller than the latency peak of the other two methods.
[0047] In summary, the method of this invention achieves better performance than existing methods during scaling. Both NativeFlink and Order-Unaware stream processing systems may experience service interruptions or severe performance degradation during system state transitions; however, the method of this invention significantly reduces the maximum processing latency during system scaling by prioritizing state transitions.
Claims
1. A method for scaling up and down stream processing jobs based on priority state transitions, comprising the following steps: Step 1: Preparation Phase: The global controller updates the task topology within the stream processing task based on the new parallelism. Then, the global controller verifies the current key space of the stream processing task with all operator instances. ; Step 2, Allocation Phase: When a stream processing task triggers a resizing operation during execution, the global controller starts a new operator instance for the stream processing task and creates new data channels for the upstream and downstream operators accordingly; then, it records the future key space of each operator that needs to update its parallelism. ; Step 3 Migration Phase: The upstream operator updates its data distribution strategy and uses a control signal-based mechanism to coordinate the state migration between operator instances. In addition, a progressive migration strategy is used to perform inter-instance state migration on operators that are being expanded or shrunk, and the state to be migrated is split into multiple parts for migration, each part being called a micro-batch. Step 4: Transfer and Cleanup Phase: Destroy instances that are no longer in use after the scaling down operation; at the same time, reclaim temporary system resources allocated during the scaling up and down process. In step 3, at the beginning of each transition phase, the global controller is responsible for injecting a control signal into the source operator to trigger a state transition. Next, the control signal can flow through the entire task topology; whenever the first operator instance When a data record is received from the input data channel, the control signal is first sent to the downstream second operator instance. If the downstream second operator instance If a state transition is required, the data distribution strategy is updated. When a state transition is required, if not all control signals have been received, the system enters the alignment state; otherwise, it enters the alignment completed state. Finally, an acknowledgment message is sent to the global controller. If the global controller has received the acknowledgment message from all parallel instances of the task, it means that a transition is complete. In step 3, during the migration of each microbatch, the global controller determines which states should be migrated based on the user-defined microbatch size. During the migration of each microbatch, the control signal contains information about all keys that need to be migrated, while each upstream instance creates a temporary routing table to determine which downstream instance to send the record to.
2. The method for scaling up and down stream processing jobs based on priority state transition according to claim 1, characterized in that, In step 1, a hierarchical data structure is used to implement state management. Based on the single-layer mapping, each key group is further divided into multiple sub-key groups. Regular stream processing is kept to run in a coarse-grained manner to avoid additional overhead. State transitions are performed in a fine-grained manner to reduce the impact on operator performance.
3. The method for scaling up and down stream processing jobs based on priority state transition according to claim 1, characterized in that, In step 2, an on-demand state access mechanism is proposed, with the first operator instance. A set of states Migration to the second operator instance The state It contains three state values, each corresponding to a key. After the state transition is triggered, the first operator instance First, the state to be migrated is sent to the second operator instance as batch data. This is called a send operation; and during this process, the second operator instance... In addition to waiting for batch data to arrive, it can also actively retrieve individual data entries, thus enabling timely processing. This is called a retrieval operation.
4. The method for scaling up and down stream processing jobs based on priority state transition according to claim 1, characterized in that, In step 4, state transfer services are provided during the migration phase based on the demand queue and response queue, and using an external key-value storage service. During the migration phase, each operator instance starts two additional lightweight threads in addition to the normal processing logic to continuously read messages from the demand queue and response queue.